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ABSTRACT 


In this thesis we discuss various issues in spatio-temporal pattern 
recognition using connectionist paradigms. We present a new back-propagation 
algorithm for training a class of neural networks for recognising patterns in 
space and time. The network we propose is a linear filter artificial neuron system. 
The basic unit uses an artificial neuron followed by a linear time-invariant or 
time-varying causal filter. Filter could be of recursive or non-recursive type. The 
neural unit uses a quadratic time-weighted output deficit error function which has 
to be minimised in the training phase. We present a learning algorithm to minimise 
the error by suitably adapting the filter parameters as well as the weights of the 
network using a gradient based error minimisation techniaue. The constraints on 
the filter parameters to ensure stability have also been derived. Computer 
simulations have been carried out extensively to evaluate the performance of the 
system on temporal and spatio-temporal pattern recognition in a variety of cases. 
Results on robustness of the network as well as convergence of the algorithm 


based on empirical observations have also been discussed. 
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Chapter 1 
Introduction 


I. Introduction 

The twentieth century has seen an unprecedented growth in information 
processing in every conceivable field of human activity. Such a growth in 
information processing has been feasible since the advent of electronic 
intruments and calculating machines in the early years of this century. With the 
advent of compact high speed electronic computers using solid-state technology in 
the sixties, more complex varieties of processing capabilities emerged. Now with 
drastic minimisation of physical dimensions of electronic ciruits using integrated 
circuit technology it has become possible to design circuits with more than a 
million "electronic switches" or transistors. This has made it practical to look at 
parallel distributed processing (PDP) techniques which were earlier considered to 
be unconventional and impractical because of the huge number of processing 
elements they need. We shall discuss PDP a little later. 

Pattern recognition, which was hitherto a forte of biological systems alone, 
soon came to be examined as one of the several information processing tasks a 
computer may handle. Pattern recognition entails a discussion on our 
conceptualization of what may constitute intelligence. Researchers in the fifties 
and sixties examined the whole issue of machine recognition of objects, and 
concurrently emerged a plethora of concepts and questions in a field that has 
since come to be known as artificial intelligence or AI. Most of the artificial 
intelligence work progressed on the Von Neumann stored -program -stored -data 
sequential computing model, in which, a program sequentially processes the data 
stored in its memory. Thus, artificial intelligence encouraged the algorithmic 
approach of manipulating object data for the purpose of pattern recognition. 



Researchers using the AI approach to pattern recognition have based their 

algorithms on theories of cognition and theories of mental representation of 

objects and concepts. These theories depend on specific models that have been 
built for explaining higher mental capacities of human beings, a couple of 
illustrative examples being spatial pattern recognition, speech recognition and the 
ability to communicate by means of a language. This is the top-down approach which 
calls for an explicit model building exercise - a model to describe what 

constitutes a particular mental process. 

Let us take for instance the task of recognising spatio-temporal patterns 
which is the topic of main interest in this thesis. Spatio-temporal patterns are 
patterns or events that occur in time and space. A specific example of such 
pattern recognition activity is that of recognizing speech. Researchers have 

developed complex models for speech and language. One such model is the hidden 
Markov model which treats speech as a doubly stochastic process. Researchers 
have used computers to perform speech recognition based on this model along with 
a model for the language in which a particular piece of speech is uttered. Using 
statistical and syntactic techniques incorporated in their algorithms, they have 
been successful in making computers perform speech recognition. 

In early fifties some researchers took an entirely different approach to 
perform pattern recognition and emulate human intelligence, an approach that has 
come to be known by a variety of names. It is the neural network or connectionist 
approach to pattern recognition. In neural network paradigms the basic 
architecture comprises of artificial neurons interconnected according to a 
predecided scheme. These artificial neurons are the mathematical descriptors of 
the biological neuron in some sense. An artificial neuron is, essentially a linear 
combiner followed by a non-linearity. These units are interconnected amongst 



themselves. Such interconnections are the mathematical analogs of the biological 
synapses between neurons, and the synaptic strengths are modelled using the 
weights of the linear combiners. The system is trained in one of the several 
available ways. During the training period the synaptic weights are modified 
suitably in accordance with a learning rule that is used with the aim of maximising 
the pattern -recognizing capacity of the system under consideration. The neurons 
in the network work in parallel, changing their state depending on their inputs. 
Such a parallel operation of neurons in an artificial neural network has put these 
paradigms into what have come to be known as Parallel Distributed Processing 
schemes or PDP, in short. 

In connectionist systems there is no explicit definition of what constitutes 
intelligence. Instead, it is entirely implicit in the synaptic efficacies or weights 
of the neural network and the particular architecture that is employed. By 
themselves, these weights individually have not yet yielded any explanation to 
researchers for the seemingly intelligent behaviour that neural networks 
demonstrate. It is only collectively that these neurons carry out the tasks that 
the system is intended to. This is then, the bottom-up approach, in constrast to the 
top-down approach emphasised by AI - a contrast between the implicit and explicit 
modes of description. 

However, connectionist notions received a severe blow in the sixties when 
Minsky and Papert £323 proved the inadequacy of the single layered Perceptron of 
Rosenblatt [473 to perform even the simplest of recognition tasks. The Perceptron 
has been proved to be incapable of synthesizing even the XOR function. But, in 
1982 Hopfield demonstrated some simple applications of recurrent networks which 
rejuvenated the interest in neural networks. In fact, the pioneering breakthroughs 
of Hopfield £223, Kohnen £263, Grossberg £163, £173, Fukushima £133, £143, £153 and 



Rummelhart and McClelland [48] in the early eighties, collectively resucitated the 
connectionist notions of pattern recognition. It was also seen that the traditional 
AI approach to pattern recognition had not yielded much success even after twenty 
odd years of research and experimentation. Further, the connectionist systems of 
Kohonen, Hopfield and others were more complex than the Perceptron and performed 
extremely well on pattern recognition tasks. That led to this renewed interest in 
neural networks. 

Objects of a pattern recognition task can be dichotomized into two 
categories namely, static patterns or objects and temporal patterns or events. 
Events are actions associated with objects, for example, the movement of a robot 
arm or spoken words in a speech recognition scheme. Much work has been done on 
both categories of patterns. However, all of traditional statistical or syntactic 
pattern recognition schemes are not only specific to the said categories of 
objects but also to the specific tasks and to the signals associated with the 
object data. While character recognition paradigms use syntax based segmentation 
and processing, speech recognition involves application of techniques like hidden 
Markov modeling - an entirely different paradigm. 

Then, the question arises as to how the brain is able to perform all such 
recognition tasks associated with various objects and events in a single framework 
of connectionist paradigms. Precisely this, has been the motivation for using 
connectionist models for pattern recognition. Besides, cognitive tasks like speech 
understanding, vision, motor control of muscular apparatus, etc. are not precisely 
defined due to their close interaction with a highly uncertain and variable 
environment, and it has proved difficult to achieve even moderately good results 
by standard syntactic or statistical techniques. Main drawback is, of course, the 
fact that diverse schemes have to be used in the traditional approach. 
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As mentioned earlier, in traditional methods of syntactic or statistical 
recognition based on AI, a model for the input is assumed, to begin with. For 
instance the picture of a human face is considered as an image with certain 
features or components like the nose, the eyes, the mouth and so on while in a 
character recognition task the images are dissected or transformed in order to 
recognise strokes, vertical and horizontal bars, closed and open curves and other 
geometric shapes. On the other hand, in connectionist paradigms such modeling, 
though useful is not always necessary. Such preliminary feature extraction is 
called preprocessing. Almost always people have sought to eliminate this phase 
because neural networks are inherently very powerful and the same system can 
often be used to recognise a human face as well as a handwritten character upon 
suitable training. 

While the preprocessing stage is kept minimal in static pattern recognition, 
most researchers have had to employ it in case of temporal pattern recognition. 
This is one thing we tried eliminating in the model we have presented in this 
thesis. 

Furthermore, if connectionist schemes of pattern recognition are inherently 
parallel, the traditional syntactic and statistical schemes are more or less 
sequential in nature and the number of dependencies can be high enough to impede 
processing considerably which results in performance degradation in many 
recognition tasks where quick responses are sought. 

II. Organisation of the thesis 

The thesis is divided into five chapters as described below. In chapter 2 we 
look at various approaches to spatio-temporal pattern recognition that many 
researchers have put forth. We also discuss some ideas regarding the use self- 



organising networks for spatio-temporal pattern analysis, especially speech 
segmentation. Finally, we discuss some ideas which led to the design of a network 
for spatio-temporal patterns, which is the main part of this thesis. 

In chapter 3 we propose a network for spatio-temporal pattern recognition 
called ALFANS, standing for A Linear Filter Artificial Neuron System. We derive 
the learning algorithm for this network and discuss the constraints on various 
parameters in ALFANS. 

In chapter 4 we present the results of our simulations of ALFAN system. We 
present some examples of the tasks that the network can handle. We draw our 
conclusions based on the results we obtained and make some critical observations. 

In chapter 5 we discuss the scope for further research in this area and 
comment on some application domains for which ALFANS is a potential candidate. 



Chapter 2 

Spatio-Temporal Pattern Recognition 


I. Introduction 

Neural networks C291 have been successfully used for a variety of tasks 
involving spatial patterns. These include tasks such as character recognition and 
regeneration, recognition of human faces, segmentation of images, image thinning 
amongst others C143, [171, [233. 

Researchers have considered neural network structures usgd for static 
spatial patterns to also recognise temporal patterns such as speech. However, 
even when the size of such networks is allowed to grow, it is often difficult to 
train static pattern recognition machines to adapt to recognise temporal 
variations [523. 

In temporal pattern recognition, in contrast to, static spatial pattern 
recognition, the entire time pattern is not available simultaneously. To overcome 
this limitation, different time samples could be appropriately delayed and stored, 
and the recognition task carried out with these stored static patterns. One 
instance of this method is the music -box approach [113. Feed-forward networks 
have often been used for speech recognition by converting temporal variations 
into static spatial patterns [63. However, the delay associated with the conversion 
and storage task may not be acceptable in some applications. 

Before we proceed to review the current state of research in neural 
networks or the so called connectionist architectures for spatio-temporal pattern 
recognition let us look at one of the traditional appraoaches that has been in use 
for speech recognition for several years now. A consideration of that method 



which is often described as the hidden Markov modeling technique, enables us to 

understand some of the key notions associated with temporal pattern recognition, 
especially of speech, and to appreciate the differences between the traditional 

schemes and the connectionist schemes. 

II. Hidden Markov Modeling of speech 

The first step in the conventional methods for temporal pattern recognition 
like speech involves explicit characterization of the input signal in terms of a 
signal model. Signal models are of great help because they not only provide a 
theoretical description of the signal but also work fairly well in practice and are 
useful in realizing prediction systems, recognition systems, identification systems 
[423 etc. In contrast, neural network paradigms do not call for explicit signal 
modeling. Any such modeling incorporated through preprocessing of input signals 
may lead to improved performance or reduce the neural network complexity. 

The signal models may be broadly divided into two categories, namely 
deterministic and stochastic models. In hidden Markov modeling (HMM) a doubly 
stochastic model [423 is assigned to the signal. It is a Markov model in which an 
observation sequence is governed by a probabilistic mapping of a state sequence 
which itself is a Markovian process that is not directly observable. 

An HMM is characterized by the following: 

1) A state space 

2) An observation space. 

3) The state transition probability distribution A. 

4) The conditional probability distribution of observations given the state B 

5) The initial state distribution n. 


The three fundamental problems of HMM [423 which form the basis of speech 



recognition are the -following: 

1) For each observation sequence 0 = 0^02 ... Of.’ anc * a 9i yen model X = <A, B, n), to 
efficiently compute P(OIX), the probability distribution of the observation 
sequence. 

2) Given an observation sequence 0, and the model X, to choose a corresponding 
state sequence Q which is optimal in some meaningful sense. 

3) To determine the model parameters X=(A, B, n> that maximize P(OIX) for a given 
observation sequence. 

Problem 1 is the analysis problem in which, given a model and an observation 
sequence, we estimate the probability distribution which led to that observation 
sequence. In problem 2 we attempt to uncover the hidden part of the model to learn 
about the structure of the model and find the optimal state sequences for 
continuous speech recognition etc. Problem 3 is the "training problem" in which we 
attempt to optimize the model parameters so as to best describe how a given 
observation sequence was generated. An observation sequence which is used to 
adjust the model parameters is called a training sequence. 

Consider a simple isolated word recognition system based on HMMs. For 
each word of a W word vocabulary we want to design a separate N- state HMM. We 
represent the speech signal of a> given word as a time sequence of a coded 
spectral vectors. Assume that the coding is done using a codebook of M unique 
spectral vectors making it an M alphabet system which means that each observation 
is the index of the spectral vector closest to the original speech in some sense. 
Hence, for each word in the vocabulary, there exists a training sequence 
consisting of a number of repetitions of sequences of codebook indices of the 
word. Firstly, we make the word models by using the solution to problem 3 to 
optimally estimate the model parameters for each model from real life speech data. 




Figure 2.1(a) Block diagram of an isolated word HMM recognizer. (Reproduced 
from 142] ) 
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Figure 2.1 (b) Block diagram of the computations required in the front end 
feature analysis of the HMM recognizer. (Reproduced from [42]) 
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This is the learning phase. We use the solution to problem 2 to segment each of 
the word training sequences into states in order to study the properties of 
spectral vectors that lead to the observations occuring in each state. This allows 
us to improve the model by increasing the number of states or change the codebook 
sizej etc., to improve its capability of modeling spoken words. Finally, using the 
solution to problem 2 we perform recognition of an unknown word to establish as 
to which among the W word models is the most likely candidate from which this 
unknown word could have most probably been generated. The fig. 2.1(a) shows the 
block diagram of the computations required in the front end feature analysis of an 
HMM recogniser, while the block diagram of an isolated word HMM recognizer is 
shown in fig. 2.1(b). 

III. Overview of literature in connectionist architectures for temporal 
pattern classification 
A. NETtalk 

With the great successes achieved in the recognition of static patterns 
using artificial neural networks, researchers started experimenting with 
connectionist architectures for spatio-temporal pattern recognition in 1986-87. 
NETtalk by Sejnowski and Rosenberg is an early effort in that direction. NETtalk 
uses a simple back-propagation network that reads a piece of English text 
character by character and reads it aloud, pronouncing it with good accuracy. 
They trained the network to produce speech sounds corresponding to a printed 
text whose characters were presented to the network one at a time, through a 
window with a fixed field of view. The field of view spanned the central letter 
alongwith 2 or 3 letters on either side of it. These extra letters were for 
providing contextual information to the network. The speech sound produced was 



based on the phonetic descriptor that was generated at any instant corresponding 
to the letter in the centre of the view window aided by other adjacent letters in 
the field of view. 

NETtalk used an heirarchical back -propagation system with three layers. 
Corresponding to each letter in the field of view there was an input network. All 
the input networks were in parallel followed by a hidden layer and an outer layer 
which generated the phoneme descripters. The training system was standard static 
back-propagation learning [481. The network at first learnt to babble like a child 
and finally, became proficient in speaking with correct pronounciations and could 
pronounce perfectly over 95*/, of the time after 50 learning cycles. Details of 
NETtalk can be found in [ill, C493. 

B. Time Delay Neural Networks (TDNN) 

The time delay neural network model was proposed by Waibel, Hanazawa et.al. 
[523. It was constructed primarily for speech learning and recognition. They have 
listed five properties that a layered feed-forward net must posses in order to be 
useful for speech recognition. Quoting them "First, it should have multiple layers 
and sufficient interconnection between units in each of these layers. This is to 
ensure that the network will have the ability to learn compfex non-linear decision 

/V 

surfaces [293. Second, the network should have the ability to represent 
relationships between events in time. These events could be spectral coefficients, 
but might also be the output of higher level feature detectors. Third, the actual 
features or abstractions learned by the network should be invariant under 
translation in time. Fourth, the learning procedure should not require precise 
temporal alignment of the labels that are to be learned. Fifth, the number of 
weights in the network should be sufficiently small compared to the amount of 



training data by extracting regularity." Their claim is that the TDNN they designed 
satisfies all these five criteria. 

Our observation is that while all these criteria must indeed be satisfied the 
amount of preprocessing used for speech learning in TDNN involving windowing the 
signal, calculating FFTs and then computing the melscale coefficients is 
substantial. It would be of interest to explore whether this "preprocessing" could 
be incorporated within the neural network itself by a suitable choice of network 
elements and architecture. 

1 . The TDNN architecture 

TDNN uses a modified form of the basic artificial neuron. Apart from the 
usual linear combiner followed by a non-linearity, Waibel, Hanazawa et.al. 
introduced delays Dj_ through in series with each input, as shown in fig. 2.2. 
Although, not stated explicitly, these delays have been interpreted as Djsz' 1 in 
the discrete time case. Each delay Dj has its own weight w^ +i . This way they gave 
each basic unit the ability to relate the current input to the past history of 
events, ie. to the inputs sampled previously in time. 

The net they used for speech recoginition consisted of 3 layers. At the 
lowest level, the input to the net comprised a set of 16 normalised melscale 
spectral coefficients. Speech sampled at 12 kHz was multiplied with a Hamming 
window and a 256 point FFT was computed. The log energies in each melscale 
energy band [533 were computed from the power spectrum, where adjacent 
coefficients in frequency overlapped by one spectral sample and were smoothed by 
reducing the shared sample by 50 percent C533. Further data reduction was 
achieved by collapsing samples adjacent in time which resulted in an overall frame 
rate of i in 10 ms. Fifteen frames of speech were collected centred around the 






manually labeled vowel onset and all coefficients of the corresponding input 
token were normalised so as to lie within +1 and -i. 
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Each TDNN unit has the ability to encode temporal relationships within the 
range of the N delays. Short duration local temporal relational features are 
formed at the lower layer and more complex longer duration features are formed 
at the higher layer. 

Learning is carried out using back-propagation. There are two phases, the 
first is the forward pass in which an input pattern is applied and the output is 
calculated, using the current set of weights. The output is compared with the 
desired target output and the error is calculated. The derivative of the error is 
propagated back through the network during the second phase and all the weights 
are adjusted to decrease the error. This is done until the desired output is 
learned with fair accuracy. 

2. Observations and comments on the TDNN 

Waibel, Hanazawa et.al. state that in the speech recognition experiment, the 
recognition rates were above 99.5 percent for 3 speakers in case of TDNN while 
recognition rates were around 96 to 99 percent in case of the best hidden Markov 
modeling technique. They state that "better classification rather than better 
generalization might be the cause of TDNNs better performance." 

TDNN is shift-invariant and does not rely on precise alignment or 
segmentation of input as the delays that have been introduced allow it to learn 
temporal connections. However, all temporal connections so formed are only upto 
order N since the maximum delay is D^. Heuristically, it is more sound to use a 
general filter model involving use of recursive filters to achieve very long term 
correlations. This is one reason why we have used a general filter in ALFAN, a 



network element described in chapter 3. 

One more observation we make is regarding the NETtalk experiment in light 
of the TDNN. NETtalk used a purely static net which scanned the text using a 5 or 
7 character wide window. In some sense this amounts to a model in which a 
dynamically varying spatial pattern is being presented and by parallely presenting 
7 characters simultaneously the temporal variation is being converted into a 
spatial pattern. Thus, they were able to relate adjacent patterns in time exploiting 
the fact that adjacent patterns in time could be represented as adjacent patterns 
in space. An extreme case could be one in which the view window spans the entire 
word, leaving no room for dynamics at all. 

One can argue that the same NETtalk experiment can be conducted using a 
TDNN and a view-window that spans a single character at a time. This is because, 
as mentioned earlier, a TDNN uses a ladder of unit delays to store time samples at 
convenient points in the network. Hence using a maximum delay of 7 units in the 
network one can possibly generate a context for the input character under 
consideration at a given time. 


C. The 'Moving Targets' training paradigm 
1. Overview 

The "Moving Targets" training algorithm proposed by Rohwer C451 is a simple 
method for training the dynamical behaviour of a neural network and is applicable 
to any training problem in discrete-time networks with arbitrary feedback. It is 
quite similar to the back-propagation paradigm, in that, the gradient of an error 
measure is used to train the network. The algorithm applies to networks with 
arbitrary connectivity not just feed-forward, hence it can be used to train 
recurrent networks for dynamical patterns according to Rohwer. 
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The key point is that hidden nodes are treated like target nodes with 
variable training data. Quoting Rohwer "These moving targets are varied during 
the minimisation process. Werbos £55] used the term moving targets to describe the 
qualitative idea that a network should set itself intermediate objectives, and vary 
these objectives as information is accumulated on their attainability and their 
usefulness for achieving overall objectives. The (coincidentally) like-named 
algorithm presented here £45] can be regarded as a quantitative realization of 
this qualitative idea." 

2. 'Moving Targets' training algorithm 

Consider a recurrent network with neuron activation values x it taken as the 
dynamical variables. Suppose x it change with time according to the following law 

*it = 22 w ij f(x j ' t-U i > 0 (2.1) 

j 

x ot = constant 

Here, w^ are weights from j to i and f is differntiable. The output is given by 

y it = f(x it ) (2.2) 

In the normal back propagation, a network is divided into input, hidden and target 
nodes. In the moving targets algorithm, analogous concepts exist but are data 
dependent instead of depending on the network architecture. Rohwer defines-' a 
node-time pair (it) as an event. He distinguishes between 3 disjoint categories of 
events, namely input I, target T and hidden events H. If X it defines the training 
data fed to the network as input, then essentially speaking 

*it ■ x it <it) el ( 2 . 3 ) 

The moving targets training method uses an actii/ation deficit error function given 
by 

w. 2 

Ead * S ( x it~ X it) 

(it) eT 


(2.4) 



Equation 2 is extended to include hidden events so that 


E = 


1 

2 


X {S w ii - X 4 

(it) e t U h ^ j ^ 


(2.5) 


This function is minimised to find a suitable set of weights. It can be seen that 
the weights depend on the moving targets X^. So one needs to compute derivative 
with respect to moving targets X as alone, where a is the node and s is the time 
instant and (as) represents an event. The details of the algorithm are given in C451 
and presented in fig. 2.3. 


Rohwer found that an output deficit quartic error funciton 

E . J ]T (10 * »t) (« it - Y it ) 4 
(it) eT 


( 2 . 6 ) 


a being a small positive constant, yielded better results because the 
quartic function chooses to enhance larger errors even if they occur 
infrequently. Further, the temporal weighting encourages the algorithm to focus 
more on late time errors than on earlier ones. This is in accordance with the 
causal nature of a real-life system. This also helps in overcoming problems of 
local minima that may be encountered. 

Rohwer tested this algorithm on three different problems — first, in which 
two inputs, taking 0,1 values, controlled the frequency of an output square wave 
for the four possible input combinations; second, in which the network learned to 
generate the time reverse of certain input time sequences; third, in which the 
network learned to respond to the second of two input pulses separated by 100 
time steps, thus demonstrating its capability to grasp distant temporal 


correlations. 
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Figure 2.3 A summary of the moving targets training algorithm 



D. The Music -Box associative memory 


The music -box associative memory is an old trick and is quite similar to 
bootstrap loading in computers. It is perhaps the crudest method to build a 
network that learns spatio-temporal sequences of patterns. Consider a 
heteroassociative memory that can recall a target pattern when shown an input 
pattern. The corresponding target is recalled because of an association between 
the input and the target. Now suppose that the sequence of patterns to be recalled 
is given as (P^,P2,P3, Pfsj ) . Let P^ be the target for the given input pattern 1 , 

P2 be the target for P^ and likewise every succeding Pi+j. be the target for the 
previous target pattern P* . Now if we make an arrangement for the generated 
target to be fed back as an input after an appropriate delay, then, given the 
trigger I, the network would recall the sequence (Pj l ,P 2,P3, ... Pj,Pi + i, ... P^j) - 

This is called the "music-box" technique because of its similarity to the 
scheme of generation of a sequence of notes by a music box. Whether this approach 
amounts to genuine learning of sequences of patterns is debatable. Just consider a 
sequence ( P P2, P3, P4, P5, P2< Ps> Pj.j ^4* P5 ••• Pn*- The music box approach would fail 
in this case because the pattern sub-sequences (P^) and (P2>P3> repeat and the 
patterns succeeding these subsequences are different at different occurrences. 
With this approach there are numerous other drawbacks as well Ciil. 


E. Single neurode systems 

Caudill and Butler have described some single neurode systems for spatio- 
temporal pattern learning [ill. An example system, described by Grossberg, using 
an outstar neurode H [see appendix B for a description of outstar] is shown in 
fig. 2 . 4 a. A triggering pulse to an outstar neurode travels along the axon A and 
the axon collaterals AC reaching the ends at different points in time due to 
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The simplest outstar capable of learning sequences. (Reproduced 



Figure 2.4 (b) The outstar connected to different synapses of the same grid 
neurodes makes a more versatile system. (Reproduced from [11]) 



Figure 2.4(c) A single outstar connected to a series of grids via a set of 
interneurons that pass the signal through. (Reproduced from [11]) 
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different positions and lengths of the axon collaterals. After training, a 
triggering input would generate a complex spatio-temporal pattern at the far ends 
of the axon collaterals AC. The signals at the ends of these collaterals [marked 
with arrows in fig. 2.4(a)! can be used appropriately for performing complex tasks 
like sensory - motor - control of a robot manipulator. 

Grossberg, however pointed out that once a trigger is received and a 
pattern gets generated, there is no way to stop it. Caudill and Butler describe 
another system shown in fig. 2.4b. It uses an outstar alongwith a set of instar 
neurodes [see appendix B for a description of instar and outstar!. The outstar is 
connected to the instar border neurodes BN as in the previous case. Here again, 
the different lengths of axon collaterals AC produce different delays at the ends. 
But each set of signals arrives at a different set of synapses on the instar 
border neurodes BN with different delays, thus creating the same effect as if the 
hub neurode H sequentially activated a series of separate outstars. On proper 
training and control of arrival times and synaptic efficacies some control can be 
achieved, via the instar neurodes, on the generated spatio-temporal pattern. 

Figure 2.4c shows another structure from Caudill and Butler which uses a 
series of outstars. A signal from the generating hub H is intercepted at varying 
times by the series of hubs I appropriately placed along the axon A. These 
secondary outstar hubs have been called interneurodes because of their 
functional similarity to the biological interneurons. The instar border neurodes 
BN in the principal and secondary outstars can communicate with other similar 
networks, hence a variety of complex spatio-temporal patterns can be generated 
upon enforcing suitable control and training. As mentioned earlier, some such 
appropriately generated patterns can be used in complex tasks requiring co- 
ordination, such as, control of robot manipulators. 



In all these networks appropriate spacing between neurodes and proper 
choice of lengths for the axon collaterals are very crucial points. One has to 
design suitable training procedures to identify the correct values for the delays. 
This is definitely not an easy job nor are we aware of any systematic enquiry or 
empirical research that may have been conducted in this connection. Further, all 
these networks are only to generate spatio-temporal patterns and not for 
recognising such patterns. Hence their utility is limited since many conventional 
methods already exist for generating spatio-temporal patterns. 


F. The outstar avalanche system 

Outstar avalanche network modifies the basic neuron used. Normally the 
activity of an outstar neurode is assumed to die instantly. However, in this 
network an exponential decay for the neurode output is assumed. The activation 
fall off is controlled by a single decay parameter called activation decay 
constant. The larger the decay constant the shorter is the response duration. 

The avalanche network consists of three layers — an input, a middle and an 
outer layer. The function of the input layer is to transmit the incoming trigger 
signal to all the neurodes of the middle layer. Hence, functionally, each input 
neurode is the hub of an outstar Csee appendix B for a description of outstar]. 

The output layer generates the desired patterns and each neurode in this 
layer is the focus of an instar beginning on neurodes in the previous layer. The 
output neurodes each have an external input through which the training patterns 
are impressed during the learning phase. The middle layer neurodes receive inputs 
from other neurodes in the same layer, as well as, from those in the input layer. 

Once trained, the avalanche network works as follows. The middle or 
avalanche layer neurodes fire only upon receiving a stimulus from the currently 



active neurode and if the previously firing neurodes in the sequence are still at 
least partially active. In other words the exponentially decaying activation 
introduces a memory into the system and is capable of exhibiting long term 
correlations. The succeeding neurodes are triggered only, if the correct 
combination of stimuli is received with proper temporal alignment. Suppose the 
middle layer neurodes are excited in the wrong order or accidentally stimulated 
with noise, any resulting spurious activity in the layer extinguishes rapidly. 
Further, for the generation of more complex patterns we can make it necessary 
that more than one neurode is used to trigger the process. 

Training this network is not difficult. First of all the avalanche layer is 
activated by presenting the desired input pattern to the network. In 
correspondence with the incoming sequence, the target spatio-temporal pattern is 
also presented to the network through the input terminals of the output neurodes. 
The neo-Hebbian learning procedure used in case of outstar learning is employed 
in this case as well. With repeated presentations the avalanche network learns the 
desired sequence of patterns. The network can learn a multitude of sequences and 
the generation of complex patterns can be controlled using the many available 
inputs to it with appropriate training. 

While all these outstar based networks can generate spatio-temporal 
patterns after appropriate training, it is probably inadequate for spatio-temporal 
pattern recognition. Robert Hecht-nielsen suggested that a Kohonen learning 
scheme can be incorporated into the avalanche layer of the network. An unknown 
sequence of patterns can be compared with a learned sequence of patterns using 
competitive learning strategy used in Kohonen's feature map generating networks. 
A good discussion of this is given in [111, 



IV. Some other approaches to spatio-temporal pattern analysis - 
Self-organising structures for temporal pattern analysis 

Self-organising neural systems have been extensively used for spatial 
pattern recognition tasks like handwritten character recognition. Notable among 
these networks are the ART 1 and ART 2 proposed by Grossberg [8],C91, the 
Cognitron [151, the Neocognitron CIS], and their variants proposed by Fukushima 
and others, and lastly, Kohonen's self -organising feature maps C271 . 

Some researchers have considered these networks as possible means of 
recognising temporal patterns like speech. One of the interesting applications is 
of speech segmentation and phoneme recognition. We have hitherto seen how the 
HMM technique is used for state estimation to carry out speech segmentaion. HMM 
requires that a model be specified for this task. HMM uses complex algorithms to 
estimate the state of the system on the basis of which speech segmentation is 
performed C421, C441 . In contrast, Self -organising neural networks, by virtue of 
their ability to learn and classify without supervision, do not require any 
information, apriori. This allows us to appropriately use these networks to 
perform speech segmentation, specifically to detect phoneme boundaries in a 
continuous speech utterance. Self -organising networks, upon adapting suitably to a 
training set, should be in a position to detect a change in pattern when one 
phoneme ends and the next begins, without any supervision . 

Assume that N consecutive samples or melscale coefficients C531 of a speech 
sequence are fed to an ART 2 or Neocognitron [see appendix C for a description of 
Neocognitron] . A process of self-organistation will take place with the repeated 
application of the input. Suppose the input speech consists of a sequence of 
elementary utterances — phonemes or other such entities. If we assume that self 
organisation process assigns a single gnostic cell to each elementary utterance 



then after training has sufficiently progressed we could expect that upon 
impressing an elementary utterance the corresponding gnostic cell would fire. As 
soon as one input pattern leads to another (a difference that is percieved by the 
network upon self-organisation) the gnostic cells would also switch 
correspondingly, thus performing a segmentation of the input sequence. The 
process of self-organisation excludes human intervention or help and makes it a 
fully automatic procedure for performing segmentation of the input sequence. The 
input can be presented as a set of N samples seen through a window of width N 
which moves M < N steps every time instant. The necessary preprocessing can 
also be done to present a set of melscale or short-time Fourier transform 
coefficients at periodic intervals. A shift-invariant network like neocognitron or 
its derivatives would be very effective in such an event because its ability to 
recognise shifted versions of a pattern would enable it to very effectively 
recognise and segment non-stationary signals like speech. 

We have tried using the neocognitron for the speech segmentation 
experiment. A program for simulating the neocognitron has been written. The 
number of layers and the number of neurons per layer are limited only by the 
computer's memory. A dynamic data structure has been used to store the neuron 
activation values. We simulated the system and trained the network to learn some 
static patterns. While the network did acquire the ability to fire the gnostic cells, 
it failed to distinguish between the two input pictures that were fed to it. This 
was the case for several pairs of pictures fed to it. The neocognitron seemed to 
be very highly sensitive to learning parameters, various thresholds and initial 
network weights. While self-organisation did take place the network did not acquire 
the ability to distinguish between two different patterns. After several efforts to 
overcome these limitations we had to abandon this line of our investigations into 
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temporal pattern recognition with neural networks. 

V. Motivation and heuristics for the conception of a Linear Filter 
Artificial Neuron System 

This section serves as a prelude to chapter 3 in which we describe a 
network called ALFANS, standing for A Linear Filter Artificial Neuron System. It 
is comprised of artificial neurons each of which is followed by a linear filter. 
Here, we discuss the issues and ideas that led us to construct ALFANS. 
Researchers have hitherto considered neural elements similar to ALFAN for 
pattern recognition tasks but we are unaware of any possible systematic study of 
such elements for spatio-temporal pattern recognition. 

» 

ALFANS uses linear filters which facilitate the introduction of finite or 
infinite duration memory into the system. Careful inspection of fig. 2.5 would 
reveal that ALFAN is similar to Grossberg's avalanche outstar and to the TDNN 
element for particular cases of the linear filter. However, it was conceived 
independently and in complete ignorance of these models only to be surprised 
later by the striking similarity between the elements of the three systems. 

It was long felt that the basic neuron structure was inadequate for spatio- 

temporal pattern learning and establishing connections in time . The first 

-t. 

modification we made was to incorporate delays z in series with each input x i( 
where z 1 denotes a unit delay (refer to fig. 2.6). The network was a simple feed- 
forward system and learning was to be via the back-propagation method. The 
weights w^- and the delays were to be learned. However, the learning algorithm 
seemed difficult to formulate in this case. Refering to fig. 2.6 we can see that 
this model is more economical than the TDNN scheme but requires a training 
procedure to modify the parameters alongwith the weights. 
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Figure 2.5 A Linear Filter Artificial Neuron 



Figure 2.6 A Time-Delay neural unit 
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Figure 2.7 A scheme employing a static feedforward network as a pre-processor 
for a recurrent network for spatio-temporal pattern recognition. 
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One idea we had was of implementing the delays z ' using neural elements 
alone. This can be achieved by constructing neural analogs of delay flip-flops 
using a simple recurrent network. This is simply done by making NAND gate 
equivalents of neural networks and then constructing a delay flip-flop. It was felt 
that perhaps a recurrent net itself could be designed using smaller flip-flop like 
networks and "neural gating schemes” to allow the system to learn a multitude of 
patterns and generate them upon receiving suitable input patterns. to trigger. To 
learn binary temporal sequences of length N it can be shown that (N+i) sequential 

N 

neural structures are needed and 2 possible sequences exist. Thus, the recurrent 
neural system would be just a counter which receives inputs and changes its state 
deterministically depending on its current state and the input at that instant. 

Making the above mentioned network recognise noisy and corrupted versions 
of a time pattern would require that the input sequences be tabulated and the 
network be constructed using the usual Boolean logic techniques. In other words, 
suppose there are P patterns to be learnt; we tabulate the P sequences each of 
length N and construct the network taking all other sequences as don't cares or as 
forbidden sequences. Then we use a static feed-forward back-propagation net to 
generate the triggering and control inputs to the recurrent network, on receiving 
a corrupted sequence of input patterns from the world outside. The scheme is 
depicted in the block diagram of fig. 2.7. In lieu of the feed-forward net one can 
use any other scheme like the ART, the Cognitron, or Kohonen's net and input to the 
recurrent network would then be the output of the gnostic cells in the recognition 
layer of these nets. 

This network is difficult and very tedious to implement. So we went back to 

**T* 

the original feed-forward network with delays z ‘ and decided to put the delays 
at the output of the neurons instead of putting them on the input side. Finally, we 



replaced the delays with general linear filters of first and second order. The 
linearity of the filters allowed us to use a back-propagation based learning 
algorithm. The linear filter artificial neuron system is presented in chapter 3 
along with the derivation of the learning algorithm. The results of our simulation 


are presented in the chapter 4. 



CHAPTER 3 


A Linear Filter Artificial Neuron System For 
Spatio-Temporal Pattern Recognition 


I. Introduction 

We have already examined the relevance of neural networks in the context of 
temporal pattern recognition in chapter 2. We have also reviewed some recent work 
done in this context. 

In this chapter we present a new back-propagation algorithm for training a 
class of neural networks for spatio-temporal pattern recognition. The basic unit in 
these networks is a linear filter artificial neuron (ALFAN) which is basically, an 
artificial neuron followed by a linear time-varying or time-invariant causal filter. 
The filter could be of recursive or non-recursive type. In this sense, essentially, 
ALFAN is a linear combiner followed by a Hammerstein predictor [103. A 
Hammerstein predictor is a non-linearity followed by a linear filter. 

Narendra and Parthasarathy [403 have talked of a network quite similar to 
ALFAN system in the context of system identification. We present the algorithm in 
this chapter while the results of our simulation will be presented in chapter 4. 


II. ALFAN structure and problem definition 

Our interest is in recognising spatio-temporal patterns using a supervised 
learning neural system. One of the promising candidates to achieve the stated 
objective with much convenience is a layered feed-forward dynamic network. By 

using the term feed-forward in the context of a dynamical system we mean, a 
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layered network in which the output of a layer never returns to its preceeding 
layers during the recognition phase, either directly or indirectly. Infact, in 
ALFAN system the output is never fedback to the input of the same neuron or even 
to the inputs of other neurons in the same layer. The structure of ALFAN element 
is as shown in fig. 3.1. The following equations describe the input-output 
relationship for ALFAN. 

K 

net(n) = ^ w^x^Cn) + w 0 (3.1) 

k=l 

0(n) = fCNet(n)] (3.2) 

y(n) = h(n)*0(n) 

N-i 

= ^ ~ h^n,m;aj f[net(m)] (3.3) 

m=0 

In the above system of equations, h(n,m;a) is a causal time-varying or time-invariant 
filter with a parameter vector a. Here, h(n,m;a) is the response at time n to an 
impulse applied at time m. A four layer feed-forward net has the structure shown 
in fig. 3.2. The input layer L is a fictitious layer and does not have any ALFAN 
elements in it. The layer K is the output layer and generates an approximation to 
the desired target patterns. 

We start with a set of inputs to the network. Each input is an N length 
sequence of real vectors. A set of P N- length sequences of real vectors 
constitute the desired targets to be generated when recognition is performed on 
the input set. When the input sequence is presented in time to the input layer the 
non-linear network must produce at the output, one of the P target sequences or a 










close approximation to it, and this must be simultaneous with the input. The linear 
filter with finite or infinite memory is used to aid in the task of generating the 
output time sequence while the non-linear neuron is used to perform a suitable 
partitioning of the input and recognition spaces to select the desired output 
pattern. 

Learning process is initiated by a suitable choice of initial neuron weights 
and filter parameters. The initial neuron weights are chosen at random and the 
filter parameters too chosen randomly, conform to some suitably specified 
restrictions that ensure the stability of the linear filter. Now suppose the 
network has K neurons in the output layer K and L neurons in the input layer L. . An 
exemplar input time sequence of length N is denoted by the vector I^ p (n) c ft*-, n 
running from 0 to N-i. The superscript p denotes the number of the input or 
target pattern and is an integer from 1 to P. The desired output sequence or 
target is a vector sequence D - 1 p <n) c . More explicitly, 

l\ 

D^ P (n) = t d P 1( ^(n) • • ■ d P K £<n> 3 . 

The output generated by layer K is 

Y^n) = C y P A £<n> ••• y p K £<n> 3. 

Our objective is to make signal the closest possible to the target and 

/ 

farthest possible from other targets D 5 -^ where p is not equal to p. 

The error minimisation between the target and the generated sequence is 
done using a least squares criterion which is used to modify the neural weights 
and filter parameters at the end of each learning cycle or at the conclusion of 
alternate learning cycles. Me chose the latter option in which we modify the 
weights keeping the filter parameters constant and then modify the filter 
parameters using the new set of weights, in the next learning cycle. 



To carry out supervised learning we define the following quadratic error 


function 
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p=i n=0 k=i ' 
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2 (n , j - l) 

<N+i> 


(3.4) 


where P is the number of exemplar patterns and N is the length of each of the 
time sequences. The weight factor is used to give more importance to errors 

in output samples occuring at later time instants. Some such scheme is appropriate 
as the system is causal and cannot be expected to recognise or regenerate 
patterns before they are applied. Further, its ability to classify or predict 
increases as more time samples arrive at the input. Our aim now would be to 
minimise E by executing a training procedure, modifying the weights of the network 
and parameters of the filter suitably. 

In the following we shall adopt the simpler notation of single subscripts 
that neural net researchers commonly employ. We represent the elements of the 
target pattern vector as d P | < (n) and the output values of layer K as yh^n) , while 
the output of the inner layer J will be written as y p j(n) and so on. Hence, we can 
write (3.4) as 


E= 


P N-i K , \2 

'll X X dB * (rt - AH 

p=i n=0 k=i ' ' 


(3.5) 


dE= p=dw + |^da 
3w 3a 

where w is the vector of all weights and a is the vector of all filter 
parameters, by forcing both partial derivatives above to zero, a point of local 
minimum for the error E in the network's parameter space can be reached. 


The recognition process, also called the testing phase, is fairly straight 



forward. We apply a vector time pattern C I^(n) Ij£(n) 1 [_l^ ^ ^ or n= ^ 
just as we apply a vector time signal to a multi-input filter. The neural net is 
updated layer by layer starting from the input to the output, for succesive time 
instants, from n= 0 to N-i. The output layer thus generates in time, a close 
approximation to the desired temporal pattern. As we have seen, the training 
procedure for the neural network is far more complex and time consuming. However, 
it is executed only when a new pattern is to be taught to the network. In contrast 
the recognition process is very fast and simple and is used to recognise patterns 
when the network is put to use. 


III. ALFAN training algorithm 
A. Learning in the output layer 

Let the change in weight w^ on learning iteration t be Awtfcj so that 

w kj t+1 “ wt kj + (3.6) 

then by choosing Aw^j to be proportional to — §IL| , 

aw kj‘t 

<3.7, 

the error E can be reduced from its value corresponding to the weights wVj ■ The 
rate of learning 7? is a positive constant and could be different for different 
layers. Similarly, 

a^t+i s a^t + Aa^t 

- -»» Sjit 0.8) 

1 . Modifying the weights 

Using (7) we have 

= -"sfc® § t K"’ - } 



(3.9) 


^ wt kj = 2^ 4w kj i,P 


where is the change associated with pattern p after learning cycle t 


Noting that 
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y p ^(n) = ^ h^<n,m :a^) f C net P ^(m) 3 
m=0 


(3.10.1) 


net P ( < (m) = ^ ^ M kj w j (m) 
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(3.10.2) 


and on omitting superscripts p and t we have 
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N-i N-i 

= 47? zz Cd^n) — y^n)) f / [net| <; (m)l h^Ov^a^) y^m) (3.11) 

m=0 n=0 


Here we have assumed that ^(n.m,^) is a causal linear filter with parameter 
vector a^. In other words 

h^(n,m;a (< .) = 0 for n< m (3.12) 

Define 


S^m) = 4 ^ ~ f'Cnet( £ -(m)] h^n/nja^) 

n^O + 


so that 


N-i 

^w^j = 7? ^ ' ^(miy^m) 
m=0 


(3.13) 


(3.14) 


2. Modifying the filter parameters 

Proceeding exactly as we did before we have 

-k 1 = Z 4 a k t ’° 
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(3.15) 



For pattern p, after learning cycle t, 


N-i 




n-0 
N-i 


(N+i> 


^p —\ 3hL»(n # m;ar,) 

Cd k <n) - y k (n)] ^ «net k (m)3 

m=6 ^ 

= 4 *a 2Z (SJi) td k (n) “ y k <n)3 Va k (n> 


n-0 


where 


drv,n,m 

v a k < n > = 2Z fInet k (m);i ai- 
m-0 ^ 


3h k (n J m,a k ) 


(3.16) 


<3.i?) 
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= 

where * denotes superposition operation. The structures shown in figures 3.3a and 
3.3b may be used to obtain the increments to the filter parameters of the outer 
and inner layers respectively. We analyse the conditions for stability in sub- 
section C. 


B. Learning in the inner layers 

Learning in the inner layers which are also referred to as the hidden 
layers, is carried out in the same manner as in conventional Back-Propagation 
algorithm for static patterns except that the algebra is more complex. The 
parameters of filters in the hidden layers must also be adapted and they, as well, 
depend on the error that is propagated inwards from the output layer. 

1 . Modifying the weights 

The weights from hidden layer I to hidden layer J are modified according 
to the following scheme employing the propagation variable $ k (m> . 




layer K. 



Figure 3.3(b) Structure to obtain parameter increments for the hidden 
layer J. 







We have 
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(3.21) 


From the definition of S k (m) in (13) we can write 


N-i N-i / \ 

4w ji = * SEE S k (m)w k j hj(m,r;aj) f'Enetj(r) y^(r) (3.22) 
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Furthermore, we define 
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so that we have 
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which identical, in form, to the expression for Aw k j 


2. Modifying the filter parameters 

The filter following the non-linearity in a layer J neuron has 
represented as hj(n;a^) . The increments Aa^t are computed next. 


(3.19) 
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We have 




N-i NJL^ 3y.(m) 

= 4r?a 22 S cSii) cdk<n) “ y k <n)3 f/[net k (m):i h k <n,m;a k )w kj 

m=0 n=D k J 


(3.25) 


N-i / / N^l \ \ 

= 4r? a 22 ] /- S (N+i) [d k (n) ~ y k <n)3 f ' Cnet k< m)] h k (n ' m ;a k ) w kj J 
m=0 ' k ' n=0 ' ’ 

3 hj(m,r;aj) 


N-i 


_ *rr— 

r=0 3 


(3.26) 


m=H ' l e 
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where 

3hi(m,r;a H ) 

v a .On) = — ^ — * fCnet,(m)] (3.28) 

0 oa^ J 

On similar lines we can proceed for the other inner layers using the propogation 
variable £<*> . So we use Sj(r) to train layer I and so on. 


IV. Choice of filter and conditions for stability 

ALFAN model makes it necessary that the filter be linear and causal. Three 
different linear time-invariant HR filters were used for simulation purposes, 
namely, 

(a) A first order filter with impulse response a n with |a| < 1 

(b) A second order resonator with imaginary poles and impulse response cos(»n) 

(c) A second order section whose poles can lie within a certain region of the unit 
circle. 

With every filter h(n;a) a set of filters dh(n,a) are associated. We must 


ensure the stability of all these filters as well. 
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A. Decaying exponential impulse response 

1. The DERIVATIVE filter 


This is the simplest of all non-trivial cases 

associated with IIR filters. 

h(n;a) = a n ; |a| < i for stability. 

(3.29) 

The Z- transform of h is 


H(z) • i— ; j- ; |z| <: i 

(i-az-i) 1 1 

(3.30) 

so that 


y(n) = a y(n-l) + fCnet(n)] 

(3 .31) 

Now 
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We make use of the following property of the Z-transform to derive the filter 

structure for : 
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From (33) and (34) we have 
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If we define 
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then 


Va<z) _ z d / z \ 

X(z) ' a dzV z-a ) 


z- 1 

i- 2a z _i + a 2 z" 2 

Hence 

(3.36) 

v a (n) = 2a v a (n-i) - a 2 v a (n-2) + f[net(n-i)3 

(3.3?) | 

I 

i 
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The filter v a <n) is shown in fig. 3.4. 

V a (z) 

2. Conditions for the stability of the DERIVATIVE filter 

To check the stability of a discrete time filter we use Jury's Test which 
states the conditions under which a filter 

N 

HCz) = ; with D(z) = ^ z"' 1 

i-0 

is stable. Appendix A provides the pertinent details. 

Clearly, the first two conditions in Jury's Test, namely, D(-i) > 0 and D(i) > 0 
are trivially satisfied for all real a. Further, on writing down the table (see 
appendix A) we get jaj ( i , which is same as the condition on h(n;a) . Hence, while 
training we can modify parameter a associated with each filter so long as it is 
within the limits prescribed above. If |at+i| > i then we freeze it at a value close 
to i , say, 0£8 , thus ensuring the stability of the filters assuming reasonable 
truncation errors. 

B. Co-sinusoid impulse response 

In this case our filter is a resonator 

h(n;«) = cos«>n 

Correspondingly, 

HKz) = -ji.z — cos» 3 
z — 2z COSa + i 

hence 

y(n) = 2y(n-i> cos« - y(n-2) + fCnet(n)] - ftnet(n-l)] cosu 


(3.38) 


(3.39) 


(3.40) 














. The DERIVATIVE fitter 


VJz) 

X(z) 


The Derivative filter must have an impulse response given by 


3h(n;a>) _ 
3(o 


-n sin»n 
.-11 


= 2 


Z ■&- 

dz 


( - 2 }) 

l z — 2z cos« + i / J 


(3 .41) 


Computing the derivative we have 


V«(z) 
X(z> ~ 


(z 4 


sine; (z - z 3 ) 
— 2z COSu + 


l ) 2 


sinto (z~^— 


z' 1 ) 


-1 -2 2 
4z cos« + z (4cos <o + 2) 


.3 _4 

4z w cos® + z 


(3.42) 


(3.43) 


Hence, we implement the filter using the following difference eauation 

2 

Ve»(n) = 4V(o<n-±) cose> — v»(n-2) E4cos « + 23 + 4v w (n-3) cos« — vo>(n-4) 

+ sin« { fCnet(n-3)l — f[net(n-i)3 ) (3.44) 

The filter for v w (n> is shown in fig. 3.5. Clearly, the poles of this filter lie on the 
unit circle and the filter is BIBO stable for any value of e>. 

C. Decaying co-sinusoid impulse response 

This is the most general of the three cases listed under section III. This is 
not only more useful but also more interesting than the other two cases 
considered hitherto. Here we have 
h(n;a,«) = a n cosun 

Correspondingly, the transfer function associated with it is given by 

H(z) = ~ . 1 cos> 3 (3.45) 

^ z — 2z a coss) + a c 

From the impulse response it is patent that for stability |a| 1 . Now we have 

the filter parameter vector a consisting of two scalars [a,«] . He have the task of 
adapting both the parameters during the learning phase. Hence, we obtain filters 
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corresponding to two different derivatives, namely, with respect to a and with 
respect to ® We utilise the filters corresponding to these while computing the 
increments Aa*- and Ato*' . 


Viz) 

1. The DERIVATIVE filter 

First we proceed with the derivation of the filter for updating the decay 


rate constant a . 

8h(n;a,«> _ _ _n-i 

mmm f I {A 

3a 

» ja n oos«n 


cos»n 


(3.46) 


( - 2 -i/ 

z (z - a cos®) 1 I 

l a dz\ 

2z a coso + ) i 


Computing the derivative 


Vafe) 

X(z) 


z a costD 


2 3 

2z a + z cos® 


( z — 2z a cos® + a ) 


2,2 


(3.47) 


-i -2 -3 2 

z cos® — 2z a + z a cos® 


-i -2 2 2 2 -3 3 -4 4 

1— 4z a cos® + z (4a cos ® + 2a .) — 4z a cos® + z a 


(3.48) 


where X(z) = ftnet(z)] as defined previously. Corresponding difference equation to 
obtain V a (n) is 

2 2 3 4 

v a (n) = 4a v a (n-i) cos® — a v a (n-2) C4cos ® + 23 + 4a v a (n-3) cos® — a v a (n-4) 

+ ftnet(n-i)3cos« — 2a f Cnet(n-2)3 + a 2 ftnet(n-3)3cos® (3.49) 


2. The DERIVATIVE filter 


Viz) 

X(z) 


In the same manner as in section Ill.C.i. we evaluate the partial derivative of 

h(n;a,») with respect to «. 

3h(n;a,®> _ _ _n _ 

— 7 T = — n a smwn 




a z sirto 


? 2 
z — 2a z cos® + a 



Computing the derivative we have 


V»(z) 
X(z) ~ 


3 3 

sin® (z a - z a) 


(z — 2z a cos« + a ) 


2 \2 


(3.50) 


sin® (z" 3 a^--- 


z" 1 a) 


-1 -2 2 2 
i— 4z a oosw -f z a <4cos « 


-h 2) — 4z~^ a^cos® -f z" 4 a 4 


(3.51) 


Hence, we implement the filter using the following difference equation 

2 2 3 

v®<n) = 4a v®(n-±) cos® — a v®(n-2) [4cos ® ■+- 23 — 4a v<,>(n-3) cos® 

+ a 4 v»(n-4) + sin# C a 3 f[net(n-3>] - a fCnet(n-i)) (3.52) 

The structure of the two filters is shown in fig. 3.6a and 3.6b. 


3. Conditions for the stability of the DERIVATIVE filters 

The stability of the two filters is critically dependent on the parameter a . 
Since both the filters have the same denominator we need to perform the Jury's 
test only once in order to determine the constraints on a. In order that the 
filter be stable the first condition in Jury's test requires that 

(i) D(i) = 1— 4a cos# + a 3 (4cos 2 # + 2) — 4a 3 cos# + a 4 > 0 (3.53.1) 

for all <o , since we place constraints only on a . 

(ii) The second condition calls for making 

(-1) 4 D<-1) = 1+ 4a cos# + a 2 (4cos 2 # + 2) + 4a 3 cos» + a 4 > 0 (3.53.2) 

Next, we write down Jury's table for the other conditions 




Figure 3.6(a) The DERIVATIVE filter to obtain in case (c). [See eqn. 3.49] 



Figure 3.6(b) The DERIVATIVE filter to obtain in case (c). (See eqn. 3.52] 
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Table 3.1 

Jury's Table 

ROW CO EFF ICgNTS 

-4a 3 cos<«> ** 

-4& COS<* 1 


- 4 * coSo> 


2 2 2 
4 * co S « + 2 * 


-4* coS« 


2 2 2 
4a cos <*> + 2k 


U-a ) 

3 2 

4a cos<*u 


4a cos<»)(a -i) 

2 4 2 

2 a u-a K 2 cos « 4 i) 


2 4 2 

2 a u-a )( 2 cos « 4 - i) 


4a coswta -i) 


3 2 

4a cosoxa -i) 

u-a 8 , 


8 2 € 2 2 
U-a ) - ±«a cos*> u-a ) 


8 a 5 cosuu-a 2 ,u-a‘ 4 )( 2 cos 2 u + 1 , 2 a 2 u-a 8 ,u-a 4 ,( 2 cos 2 »+i, 

- 4(i-a 8 )u-a € )» cos» - i€a 4 u-a € )(i-a 2 )co5» 


From the table the other constraints on a ■ are 


(iii) 1 > |a 4 | 

(iv) |i-a 8 | > | 4a 3 cosa>(a 2 -i) | 

(v) | (i-a 2 ) - i6a 6 cosc(i-a 2 ) 2 j > 

2a 2 | <i-a®)(l-a 4 )(2cos 2 ft'+i) - Sa 2 (i-a 6 )(i-a 2 )cos 2 *> | 

Conditions <i), (ii), (iii) directly yield Jaj < i . Looking at (iv) for 

i-a 8 > 4a 3 (i-a 2 ) 

or i s a 3 (i-a 2 ) 

4 i-a 8 

which yields a {- i . For a < 0 
i-a 8 > 4a 3 (i-a 2 ) 

which once again yields a >= -i . 


(3.53.3) 

(3.53.4) 


(3.53.5) 

a > 0 we have 


Finally, on looking at (v) we can write the following stronger inequality 
| (i-a 8 ) 2 - i6a 6 (i-a 2 ) 2 | > 2a 2 | <i-a 8 )(i-a 4 ) - 8a 2 (i-a 6 )(i-a 2 ) | (3.54) 

which finally yields |aj <0.6 approximately on solving numerically. This is probably 
a conservative estimate. However, from our simulation we found that it is very 
close to the exact limit on a . Since, in practice, our signals are finite duration 



and bounded, the filter for v a (n) and v«(n) never become unstable. However, if 


|a| > 0.6 

then for large enough time instants the values of v a (n) and v»(n) become very 
large, hence creating finite precision and other computational problems. Keeping 
this in mind we limited the value of |a| to a maximum of 0.7 and most of the time 
to less than 0.6. 

We present the results of our simulation in the next chapter. 



Chapter 4 
Simulation Results 


I. Introduction 

ALFANS was designed with the aim of performing spatio-temporal pattern 
recognition. In order to test how well this network could perform the said task, a 
number of computer simulation experiments were designed and carried out. Each 
experiment was designed to see if the network could fulfil some elementary 
objectives which collectively define some aspects of spatio-temporal pattern 
recognition. We present these experiments, first defining the objectives and then 
describing the results. The experiments are discussed in the increasing order of 
their complexity. 

We also present the results of some experiments we performed to look into 
certain other aspects of network behaviour and characteristics of the learning 
algorithm, like convergence, stability, speed of learning, and the kinds of patterns 
the network is capable Cor incapable) of recognising. 

II. Learning experiments with single patterns 
A. Objectives 

The main objectives are listed below. 

(a) To examine whether the network has the capacity to learn a single pattern 
using the rule derived in chapter 3. By learning we mean, here, the ability of the 
algorithm to adjust the network weights to minimise the error E "sufficiently." The 
term "sufficiently" can be quantified using ah acceptability criterion. In our 
experiments we assumed that a reduction in error E by 50% per pattern was 



"sufficient" to claim that the network had learnt the input-target map or the input 
pattern. The value of 50% is on the higher side and the choice was motivated after 
noting the reduction in error E in a large number of recognition experiments 
involving many pairs of input-target .patterns. 

(b) The second objective was to test if the network could acquire the ability to 
perform non-linear prediction of the target pattern "sufficiently" well. Once again 
we use an acceptability criterion to say that the prediction error based on 
equation 3.5 must be less than about 10“/. of the value at start for that pattern. 
The value 10% is based on common experience and can be linked to the signal to 
prediction noise ratio as well as the rate-distortion function in some way. 

(c) It is also important to examine whether the learning algorithm is stable and 
convergent, ie. whether the weights and parameters of the network stabilise, and 
the conditions under which this may be achieved. 

(d) To examine what kind of patterns a network of this sort can learn. 

B. Observations and Discussions 

In temporal pattern recognition experiments we trained a 4 layer network 
with i neuron in the 4th or output layer, 5 in the 3rd and 2 in the second layer. 
The first layer being the input layer had no neurons [see fig. 3.23. There was one 
input node. The rationale for this choice lay in the amount of computer time we 
could spend in getting a reasonably successful test performance. Too large a 
network means too much time spent on a single learning cycle while too small a 
network could lead to loss in performance or inhibit learning. The layer previous 

to the output layer was chosen to have a large enough number of neurons to 

# 

improve the prediction performance of the network since the output unit performs 



some kind of a synthesis using the signals from the previous layer which has five 
neurons in our simulations . 

The activation function for the inner layers was chosen as 


f <x> = 


(4.1) 


1 + exp(-gx) 

where g is a constant that defines the hardness of the non-linearity which after 
trial and error was given a value 0.7 for satisfactory performance on recognition 
experiments without the net going into saturation every now so often. 


For the outer layer, however, the activation function was chosen as 


f (x) = 


(4.2) 


1 + exp(-gx) 

where 1 is a constant and defines the level of the output. This was incorporated in 
order to train patterns which were not normalized. Level was chosen as 4.00 after 
trial and error. 


In the experiments described below simulations were carried out using 
linear second order filters with decaying co-sinusoidal impulse responses 
corresponding to case (c) in section 3.IV.C. 


We now present the results of 6 experiments chosen out of the many that 
were performed. Tables 4.2 (a) through 4.2 (e) list the learning rates 7), 1? u and 7? a 
used in the simulation programs. 


layer 

Neurons 

T) 

T) w 


4 

l 

0.1 

0.1 

0.1 

3 

5 

0.1 

0.1 

0.1 

2 

2 

0.1 

0.1 

0.1 

i 

i 

- 

- 

- 


Table 4.2 (a) Learning rates used in experiments of sections II and III- 



layer 

Neurons 

V 



4 

i 

0.08 

0.03 

0.03 

3 

5 

O.i 

0.05 

0.05 

2 

2 

0.2 

0.05 

0.05 

1 

1 


- 

_ 

L2 (b) Learning rates used in experiments of section 

layer 

Neurons 

V 



4 

2 

0.02 

O.Oi 

O.OOi 

3 

5 

0.05 

0.02 

0.005 

2 

2 

O.i 

0.02 

0.008 

i 

2 

_ 

- 



Table 4.2 (c) Learning rates used in experiments of section II and 


layer 

Neurons 

T) 



4 

i 

O.OOOi 

0.0002 

o.oooooe 

3 

5 

0.00005 

0.0002 

0.000005 

2 

2 

0.00002 

O.OOOi 

0.000001 

1 

i 

- 

- 

_ 


Table 4.2 (d) Learning rates used in experiments of section II and III. 


layer 

Neurons 

V 

fylJ 


4 

i 

O 

1 

iO' 6 

8xl0~ 7 

3 

5 

5xi0' 7 

2xi0' 6 

5xiO~ 7 

2 

2 

iO -6 

2xi0' 6 

2xiO~ 7 

i 

i 

- 




Table 4.2 (e) Learning rates used in experiments of section II and III. 



1 . Experiment 1 

This was the first experiment we performed with temporal patterns. For 
simplicity we chose sinusoids or their linear combinations as the input and target 
patterns . 

Input : insine 1 : 0.8sin(0.4n) 

Target : outsinei : 0.8sin(0.4n) + 0.4sin(D.8n) + Q.24sin(i.2n) 

Test inputs : 

til ibsinel : 0.8sin(0.4n) + Unoise(n) 

C21 insine2 : 0.8sin(0.6n) 

C33 insine3 : 0.8sin(0.5n) 

where Unoise(n) is zero mean uniform noise of variance ~ over C-0.5, +0.53. These 

patterns are shown in figures 4.1(a) through (c). 

Error E at start = 44.65 

Error reduced to 10.332 on the 96 learning cycle. 

Thereafter, the error increased drastically and fluctuated randomly between 
438 and 1466. However, when the learning rates were reduced considerably to those 
shown in table 4.2(e) the weights and parameters (see table 4.2.1) showed little 
change and the error E decreased steadily, empirically suggesting that the 
algorithm has convergent property for low enough learning rates although it is 
divergent for large values of W, T? u and f/ a . This is quite in accordance with the 
fact that in the derivation of the algorithm the increments in weights and 
parameters were assumed to be infinitesimal changes because they were made 
proportional to the derivative of the error E. What remains unknown are the 
bounds on the learning rate constants and other conditions which can guarantee 


convergence. 




Figure 4.1 (a) Input: insinel : 0.8sin(0.4n) 



Figure 4.1(b) Target : outsinel : 0.8sin(0.4n) + 0.4sin(0.8n) + 0.24sin(l .2n) 



Figure 4.1 (c) Noisy pattern : ibsinel : 0.8sin(0.4n) + Unoise(n) 
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Figure 4.2 (a) Input: insine2 0.8sin(0.6n) 



Figure 4.2(b) Target .- outsine2 : 0.8sin(0.6n) + 0.24sin(1.2n) + 0.32sin(1.8n) 



TIME 

Figure 4.2(c) Noisy pattern : ibsine2 : 0.8sin(0.6n) + Unoise(n) 


i 
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Figure 4.3(a) Input : insine3 : 0.8sin(0.5n) 



TIME 

Figure 4.3(b) Target .- outsine3 : 0.8sin(n) + 0.32sin(1.5nj 



Figure 4.3(c) Noisy pattern : ibsine3 : 0.8sin(0.5n) + Unoise(n) 
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Figure 4 .4 (a) Input : insine4 0.8sin(0.3n) 



Figure 4.4(b) Target: outsine4 : 0.8sin(0.3n) 



TIME 

Figure 4.4(c) Noisy pattern .- ibsine4 : 0.8sin(0.3n) + Unoise(n) 


l 

I 


AMPLITUDE 



Figure 4.5(a) Input : insineS : 0.8sin(0.3n) + 0.6cos(0.4n) 



Figure 4.5(b) Target : outsine5 : 0.8sin(0.3n) + 0.8cos(0.4n) + 0.4cos(0.6n) 
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Figure 4.6(a) Input: insine6 : 0.8sinl0.6n + O.lcos(O.ln)] 



TIME - 

Figure 4.6(b) Target outsine6 : 0.8sin(0.6n) + 0.6cos(1.2n) + 0.3cos(1.8n) 



Figure 4.6(c) Noisy pattern : ibsine6 : 0.8sin[0.6n + O.lcos(O.ln)] + Unoise(n) 



1.00 
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— 2.00 ~ j — r r ~ T~rT~T i i i | • i i i i i rrrr ] 
0.00 50.00 100I00 

TIME 

Figure 4.7(a) Input: insine7 : 0.8cos[0.7n + O.lcos(O.ln)] 



Figure 4.7(b) Target: outsine7 : 0.8sin(0.7n) + 0.5cos(1.4n) + 0.3cos(2.1n) 



TIME 

Figure 4.7(c) Noisy pattern : ibsine7 : 0.8cos[0.7n + O.lcos(O.ln)] + Unoise(n) 



Error = 10.332334 

II 12 J1 J2 J3 J4 JS K1 

u 0.79151 -5.81689 3.71686 2.12528 1.48737 0.23398 0.81328 0.60261 
a -050000 0.50000 -050000 -0.17758 0.02157 0.49924 034089 -050000 


w „ 1.96172 
I 

w .. 0.75392 0.93801 

Jl 

0.23913 0.17806 
W, . 18.97129 0.07855 

kj 


WEIGHTS 

137314 

024801 0.08726 025352 0.11491 

023720 0.12022 

0.09317 0.08700 0.08666 


Table 4.2.1 Weights and filter parameters of experiment 1 (section II) 

Further, we may note that the prediction error is too large and can 
conclude that for this choice of linear filter and learning rates the network is 
not a good predictor in this case. However, the network fulfils the first objective. 
This can be easily verified by feeding a noise corrupted version of insinei. We 
used zero mean uniform noise of variance — in the interval [-0.5,0.53 to corrupt 
insinei. This corresponds to a signal to noise ratio (SNR) of roughly 5.8 dE which 
is fairly low. Yet, upon testing with the corrupted pattern ibsine i, the network 
produced an error E = 12.148 which is very close to the value that was obtained 
using insinei and is still less than | of the error at start, i.e. 44.65. 

The same network, on feeding another input pattern insine3 yielded a very 
large error E = 64 55 and on input pattern insine2 it yielded E = 74.30 which is 


again very high. 



This suggest that the network could be trained to distinguish a class of 


patterns from all others. This can be also seen in the experiments that follow. 

2. Experiment 2 

In this experiment we trained the network to recognise a narrow band FM 
wave. The target pattern was chosen to have a component with the same frequency 
as that of the carrier of the input FM wave. The weights and filter parameters 
obtained on training are shown in table 4.2.2. 

Input : insine6 : C3.8sinC0.6n + O.icos(O.in)] : A narrow band FM signal, 
target : outsine6 : 0.8sin(0.6n) + 0.5cos(1.2n) + 0.3cos(i.8n) 

Test inputs : 

C13 ibsine6 : 0.8sinC0.6n + O.icos(O.in)] + Unoise(n) : Noisy input 
which is centered around the carrier of the input FM wave. 

123 insine2 : 0.8sin<D.6n) 

Error at start = 50.259 
Error reduced to 17.55 

It can be seen once again that although the target pattern prediction is 
poor the extent of learning is quite satisfactory. On feeding the noise corrupted 
version of insineS the error produced is E = 22.012 which is close to what was 
produced using the training pattern insine6. 

What is most interesting is that on testing with pattern insine2 the network 
yielded an error E = 17.204 which is less than the value obtained using the 
original FM wave insine6. A convincing explanation for this phenomenon is that the 
unmodulated tone insine2 is closer to the actual exemplar of the class the network 
learned to distinguish. The FM wave can be considered to have been treated as a 
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Error = 17.554953 



11 

12 

Jl 

J2 

J3 

J4 

J5 

K1 

u 

1.60777 

11)9986 

1.54627 

2.11377 

1.48761 

024973 

0.82746 

3.44014 

a 

0.50000 

0.50000 

-0.50000 

-0.12355 

0.00398 

0.25264 

0.14498 

-0.50000 





WEIGHTS 




w n 

1.67141 



1.05147 





u 

W:- 

2.64912 

0.65412 


0.20430 

0.07117 


0.20551 

0.09696 

Jl 

0.17008 

0.15486 


0.19639 

0.10179 




hi 

7.86779 

0.07855 

0.09317 


0.08700 

0.08666 


Table 4.2.2 Weights and filter parameters of experiment 2 (section II) 

noisy version of the exemplar where a small random phase variation has been 
introduced. This illustsrates the fact that the network is a frequency selective 
network more or less and classifies patterns based on the input and target 
pattern frequencies. This fact will be further exemplified in some of the 
experiments that follow where the network refuses to learn maps between input and 
target which have no common frequency components or other characteristics. 


3. Experiment 3 

The results of this experiment illustrate the prediction capabilities of this 
network. The weights and parameters of the trained network are shown table 4.2.3. 
Input : insine4 : G.BsinCO ,3n) 

Target : outsine4 : 0 .8sin(0.3n) 

Test input : ibsine4 : 0.8 sin(G.3n> + Unoise(n) 



where U noise(n) is once again zero mean uniform noise in the interval [-0.5.0.53 
with variance ^ . 

Error at start = 32.038 

On testing with noisy incut ibsine4 error was E = 0.2614 which is quite small 
compared to the starting value E = 32 .038 . This shows that if the algorithm is a 
converging one (we have no theoretical justification to claim its convergence or 
divergence characteristics) then for low enough learning rates the error E would 
steadily decrease to a value E TO as number of learning cycles become very large. 
It is then obvious that the system would predict at best with an error E^ and no 
less. The value of E^ in this case, is probably close to zero. Hence the network 
proved to be a very good predictor here and the error E, after training, is within 
10‘/, of its value at start, as stipulated in section II .A. More about prediction is 
discussed in some experiments that follow. 


Error = 0.26M 

II 12 J1 J2 J3 J4 J5 K1 


u 0.79151 -5.81689 3.71686 2.12528 1.48737 0.23398 0.81328 0.60261 
a -0.50000 050000 -0.50000 -0.17758 0.02157 0.49924 024089 -0.50000 


w 1.96172 
I 

w .. 0.75392 0.93801 

Jl 

J 023913 0.17806 

Wl . 18.97129 0.07855 

kj 


WEIGHTS 

1.37314 

024807 008726 025355 0.11490 

023720 0.12022 

0.09317 008700 008666 


Table 4.2.3 Weights and filter parameters of experiment 3 (section II) 
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4. Experiment 4 

This experiment illustrates the fact that the network is a frequency 
selective one. So far we chose the input and target patterns with the restriction 
that there be at least one component sinusoid in the target that has the same 
frequency as the predominant component in the input. In this experiment we chose 
the target pattern to contain only the higher harmonics of the input sinusoid as 
described below. 

Input : insine3 : 0.8sin(D.5n) 

Target : outsine3 : O.Ssin(i.On) + 0.32sin(i.5n) 

Error at start = 37.090099 

Error reduced to 37.090061 in 1446 iterations on using the learning rates 
shown in table 4.2(d). One can see that the network hardly learnt at all. We carried 
on learning for another 378 iterations using the rates shown in table 4.2(e) but 
the error did not reduce at all. 

This suggests that the network cannot learn maps between patterns with 
widely differing characteristics. The same conclusion can be drawn from the 
results of experiment 6 to be discussed later. 

5. Experiment 5 

In this experiment we shifted to another class of input and target patterns. 
Here, we used decaying exponentials to train the network and it was quite a 
success. The weights and filter parameters are shown in table 4.2.4. 

Input : exp2 : 0.8exp(-0.in) 

Target : expout2 : 1.2exp(-0.2) + 0.6exp(-0.4n) 


Error at start = 0.425888 
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Error reduced to 0.092013 after 752 iterations. 

This shows that the network is not only capable of learning sinusoidal 
patterns but can also learn patterns like decaying exponentials provided the input 
and target have common characteristics . 


Error = 0.092013 

11 12 J1 J2 J3 J4 J5 


K1 


u 3.82364 0.68382 8.50577 2.11672 1.48763 024490 0.82421 2.35365 

a 0.50000 -0.50000 -0.50000 -0.13502 0.01379 0.36272 0.17379 -0.50000 


W il 

W 


Jl 



0.55522 

4JD2528 0.76624 

0.19234 0.14758 

16.79617 0.07855 


WEIGHTS 

1.13138 

021471 0.06320 021569 058727 

0.19915 0.09316 

0.09317 0.08700 0.08666 


Table 4.2.4 Weights and filter parameters of experiment 5 (section II) 

6. Experiment 6 

In this experiment we chose the input and target from two different classes 
to illustrate that learning is unlikely to occur in the first place. The input and 
target hardly have any common characteristics here. 

Input : expl : 0.8exp(-0.2n) 

Target : expoutl : 0.8sin(0.6n) + 0.4cos(i.2n) + 0.02sin(3n) 


Error at start = 47.678257 



The error remained at this value even after 2i3 iterations when training was 
carried out using the learning rates shown in table 4.2(d). 

Thus, it seems empirically that the only criterion for learning to be 
successful in case of ALFANS is that both input and output patterns must have 
some common characteristics . Incase of mixtures of sinusoids or square waves as 
inputs, the targets must have at least one of its components matching in frequency 
with one of the components of the input, as mentioned earlier. Experiments with 
square waves were also carried out and some of them have been discussed in 
section III which deals with pattern recognition. Our experiments also suggest that 
target patterns must be of a low pass nature with not too many higher harmonics. 
This is because prediction of abrupt changes in pattern with time is very difficult 
due to the low pass nature of the network. 

Since single pattern learning experiments were by and large successful and 
yielded much insight into the network behaviour, learning process, and the 
suitability of certain patterns as inputs and targets, we proceeded to carry out 
pattern recognition experiments in which the network was trained to recognise and 
classify the input patterns. Proper prediction of the desired target could not be 
guaranteed as we discovered in the experiments described so far. 

Ill Temporal pattern recognition experiments 

In these experiments the objectives (a) through (d) listed in section II .A 
remain as important. Apart from these there is one singularly most important 
objective, namely to see whether the network can recognise an input pattern to be 
belonging to one out of a number of class that it has been trained for. By and 
large we found that if the pattern pairs could be taught to the network 



individually as in section II the network could also perform recognition on these 


pattern pairs fairly well. 

We shall first discuss temporal pattern recognition with single input-single 
output ALFANS. Then we shall discuss spatio-temporal pattern recognition in which 
the network has more than 1 node at both the input and the output. 

1 . Experiment 1 

In this experiment we chose just two pattern pairs both of which could be 
taught to the network, individually. 

Inputs : 

til insinei : 0.8sin(0.4n) 

C23 insine2 : O.Bsin(Q.6n) 

Targets : 

til outsinel : 0.8sin(0.4n) + 0.4sin(0.8n> + 0.24sin(1.2n) 

[21 outsine2 : 0.8sin(0.6n) + 0.24sin(i.2n> + 0 .32sin(i .8n) 

Test inputs : 

[13 ibsinei : 0.8sin(Q.4n) + Unoise(n) 

[23 ibsine2 : Q.8sin(0.6n) + Unoise(n) 

[33 insineS : 0.Bsin[0.6 + 0.1cos(0.in>3 

On training with insinei : outsinel and insine2 : outsine2 using the learning 
rates in table 4.2(b), 

Error at start = 84.605 

Error reduced to 17.463 in the 114th learning cycle. 

Thereafter, the learning rates were reduced to those shown in table 4.2(c). Error 
further reduced to 14.246 and stabilised around that value for very low learning 



71 


rates shown in table 4.2(e). The learned weights and filter parameters are shown 
in table 4.3.1 while figures 4. 1(a) -(c), 4. 2 (a) -(c) and 4. 3 (a) -(c) show these patterns. 

Then, we performed recognition tests usings the trained network. The 
results of the recognition experiment are tabulated in table 4.3.2. The first column 
shows the input fed to the network, the second column holds the target with which 
the output is compared, the third column shows the error E between the target and 
generated output. The target for which the error E is minimum, given the input, is 
the class to which that input belongs. 

It is clearly seen that on feeding insinel the error E between the generated 
output and the target outsinei is much less than the error when outsine2 is 
specified as the target. This is again the case on feeding ibsinel which is same as 
insinel corrupted with additive uniform noise of mean zero and variance in the 
interval [-0.5, 0.5], except that the error increased from 9.17 to 15.01, which is 
still quite small when compared to 72.74 which is the error E between output and 
outsine2 with ibsinel at the input. 


Error = 14.236181 

II 12 J1 J2 J3 J4 JS K1 

u 1.79858 028189 91.3500 2.11522 1.49123 024862 0.82828 2.90972 

a -087537 -0.5000 -0.17635 -0.05913 -080642 0.11017 0.06117 0.46169 


v, 4.75082 

W .. -1.57750 0.45769 

JJ 

0.45752 0.30029 

w. . -1380250 0.07855 

kj 


WEIGHTS 

1.27432 

0.45962 020140 080480 024955 

0.46644 024458 

089317 088700 0.08666 


Table 4.3.1 Weights and filter parameters of experiment 1 (section III) 
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S.no. Input 

Target 

Error 

Class 

i. insine i 

outsinei 

9.17 

outsinei 


outsine2 

70.59 


2 . insine2 

outsinei 

75.42 



outsine2 

5.07 

outsine2 

3. ibsinei 

outsinei 

15.01 

outsinei 


outsine2 

72.74 


4 . ibsine2 

outsinei 

76.56 



. outsine2 

9.86 

outsine2 

5. insine6 

outsinei 

76.79 



--.gutsioeZ 

6.16 

<?ytgjn.g2_ 


Table 4.3.2 Results of experiment 1 (section III) 


The pattern insineG belongs to the second class as shown in table 4.3.2. 
This is once again as expected because insineS is an FM wave with carrier 
frequency a) = 0.6 which is also the frequency of the single tone insine2 for which 
outsine2 is the desired target. Thus, insineS can be treated as insine2 with a 
small random term added to the phase. 


2. Experiment 2 

In this experiment we chose 3 pairs of patterns as listed below. 
Inputs : 

[13 insine i : 0.8sin(0.4n) 

[23 insine2 : 0.8sin(0.6n) 

[33 insine4 : 0.8sin(0.3n) 

Targets : 

[13 outsinei ■' 0.8sin(0.4n) + 0.4sin(0.8n) + 0.24sin(i.2n) 

[23 outsine2 : 0.8sin(0.6n) + 0.24sin(i.2n) + 0.32sin(i.8n) 



C3] outsine4 : 0.8sin(0.3n) 


Test Incuts : 

E13 ibsinel : 0.8sin(0.4n) + Unoise(n) 

C2] ibsine2 : 0.8sin(0.6n) + Unoise(n) 

[33 ibsine4 : 0.8sin(0.3n) + Unoise(n) 

[43 insineS : 0.8sinC0.6n + 0.icos(0.in)3 

[53 insqrl : sgn[ insine i(n) 3 

[63 insqr2 : sgn[ insine2(n) 3 

[73 insqr4 : sgn[ insine4(n) 3 

[83 insqr6 : sgnC0.8sirl0.6n + 0.5cos(0.in)3> 

Table 4.3.4 lists the results of the recognition experiment while table 4.3.3 holds 
the weights and filter parameters obtained on training the system. Input and target 
patterns are shown figures 4. 1(a)- Co) through 4. 7 (a) -(c). In this experiment we also 
fed the four square waves listed above as test inputs and watched the excellent 
classification that the network performed. 


Error = 29.387571 

II 12 J1 J2 J3 J4 JS Kt 

w -2.76174 -155511 023410 2.12135 1.48814 024614 082035 -126787 

a -0.50000 0.50000 050000 -0.17854 0.02128 0.43928 023446 -0.50000 


w., -1.17759 

ll 

w .. -0.99492 104034 

II 

° 028564 0.18915 

w. . 1938893 0.07855 

kj 


WEIGHTS 

084809 

028052 009615 029017 0.12512 

027426 0.13018 

0.09317 008700 008666 


Table 4.3.3 Weights and filter parameters of experiment 2 (section III) 
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We can clearly see how the network is able to recognise the square waves 
to be belonging to the appropriate classes. Intuitively, a square wave derived from 
insine i is closer to insinel than to any other sinewave, especially since that 
square wave is composed of harmonics of only one single tone, namely insinei. The 
network distinguishes and classifies the input just on the basis of this fact. 


S.no. 

Input 

Target 

Error 

Class 

1 . 

insinei 

ouisinei 

13.84 

outsinei 



outsine2 

79.00 




outsine4 

88.87 


2. 

insine2 

outsinei 

78.78 




outsine2 

12.61 

outsineZ 



outsine4 

81.75 


3. 

insine4 

outsinei 

107.73 




outsine2 

96.28 




outsine4 

2.94 

outsine4 

4. 

ibsinei 

outsinei 

14.15 

outsinei 



outsine2 

81.54 




outsine4 

97.87 


5. 

ibsine2 

outsinei 

77.51 




outsine2 

15.22 

outsine2 



outsine4 

90.16 


6. 

ibsine4 

outsinei 

97.99 




outsine2 

88.30 




outsine4 

5.88 

outsine4 

7. 

insineS 

outsinei 

81.14 




outsine2 

14.09 

outsine2 



outsine4 

81.31 


8. 

insqri 

outsinei 

29.81 

outsinei 



outsine2 

122.52 




outsine4 

146.42 


9. 

insqr2 

outsinei 

124.49 




outsine2 

27.88 

outsine2 



outsine4 

125,34 . . 


10. 

insqr4 

outsinei 

171.00 




outsine2 

154.49 




outsine4 

29.66 

outsine4 

il. 

insqr2 

outsinei 

135.18 




outsine2 

43.91 

outsine2 



outsine4 

128.21 



Table 4.3.4 Results of experiment 2 (section III) 



3. Experiment 3 

This experiment was performed using 4 pairs of input and target patterns. 
Two of them were single tones and two were FM waves as listed below. 

Inputs ■' 

Cil insine i : 0.8sin(0.4n) 

[21 insine4 : 0.8sin(0.3n) 

[31 insine6 : 0.8sinE0.6n + O.icos(O.in)] 

[41 insine? : 0.8sin[0.?n + O.icos(O.in)] 

Targets : 

Cil outsinei : 0.8sin(0.4n) + 0.4sin(Q.8n) + 0.24sin(i.2n) 

[23 outsine4 : D.8sin(0.3n) 

[33 outsineS : 0.8sin(0.6n) + 0.5cos(i.2n) + 0.3cos(i.8n) 

[43 outsine? : 0.8sin(0.7n> + 0.5cos(i.4n) + 0.3cos(2.in) 

Test inputs : 

Ci3 ibsinei : insine i(n) + Unoise(n) 

[23 ibsine4 : insine4Cn) + Unoise(n) 

[33 ibsineS : insineS(n) + Unoise(n) 

[43 ibsine7 : insine7(n) + Unoise(n) 

[53 insine2 : insine2(n) + Unoise(n) 

Error at start = 175.21 
Error reduced to 69.90 

The results of the experiment are shown in table 4.3.5 while the weights are 
and filter parameters obtained on training are shown in table 4.3.6. The figures 
4. 8 (a) -(d) show the generated outputs and how well they predict the target. As can 
be seen in these figures, the generated outputs are perfectly in phase with the 



S.no. 

Input 

Target 

Error 

Class 

i. 

insine 1 

outsinei 

16.72 

outsinei 



outsine4 

49.08 




outsineS 

56.58 




outsine? 

64.76 


2. 

insine4 

outsinei 

64.32 




outsine4 

4.72 

outsine4 



outsinefe 

64.73 




outsine7 

66.39 


3. 

insineS 

outsinei 

51.67 




outsine4 

46.20 




outsineS 

23.46 

outsine6 



outsine? 

66.39 


4. 

insine7 

outsinei 

60.95 




outsine4 

40.05 




outsineS 

61.77 




outsine7 

25,00. . 

outsine? 

5. 

ibsinei 

outsinei 

15.00 

outsinei 



outsine4 

52.62 




outsine6 

57.13 




outsine7 

69.13 


6. 

ibsine4 

outsinei 

59.99 




outsine4 

7.04 

outsine4 



outsineS 

63.01 




outsine? 

64,00 


7. 

ibsineS 

outsinei 

49.59 




outsine4 

49.65 




outsine6 

24.27 

outsineS 



outsine7 

70.50 


8. 

ibsine7 

outsinei 

56.94 




outsine4 

41.71 




outsine6 

60.14 




outsine? 

27.96 

outsine? 

9. 

insine2 

outsinei 

50.71 




outsine4 

46.38 




outsineS 

22.95 

outsine6 



outsine7 

67.02 



Table 4.3.5 Results of experiment 3 (section III) 

predominant component of the target. However, the prediction error E for each 
pattern was more than the acceptable limit of 10% as stipulated in section II .A. 
What is interesting is that the outputs generated by input patterns 1 through 4 do 
not at all match any targets other than the corresponding ones as can be seen in 
figures 4. 8 (a) -(d) and 4. 9 (a) -(d). Further, on feeding the noisy test patterns i 
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through 4 shown in figures 4.1 through 4.7, the generated outputs are clean 
sinusoids with their phases matching the phases of their corresponding targets. 

This can be seen in figures 4.10(a)-(d). 


Error = 69.903511 

II 12 J1 J2 J3 J4 J5 K1 

u 1.56039 1.65396 3.13122 2.11468 1.48714 024715 0.82598 4.91838 

a 026609 088985 -0.50000 -0.11500 0.00575 026770 0.14011 -050000 


WEIGHTS 


1.19647 


0.86546 


1.76212 

0.57325 

0.19333 086296 

0.19156 0.08712 

0.16461 

0.14679 

0.17509 0.09290 


8.76608 

0.07855 

0.09317 0.08700 

0.08666 


Table 4.3.6 Weights and filter parameters of experiment 3 (section III) 
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Figure 4.8(a) Target outsinel and the output due to insinel 


I 
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Figure 4.8(b) Target outsine4 and the output due to insine4 



Figure 4.8(c) Target outsine6 and the output due to insine6 



Figure 4.8(d) Target outsine7 and the output due to insine? 




0.00 50.00 100.00 

TIME 

Figure 4.9(a) Target outsine4 and the output due to insine6 
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TIME 

Figure 4.9(b) Target outsine6 and the output due to insinel 
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Figure 4.9(c) Target outsine6 and the output due to insine-4 
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Figure 4.9(d) Target outsinel and the output due to insine? 
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Figure 4.10(a) Target outsinel and the output due to ibsinel 
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Figure 4.10(b) Target outsine4 and the output due to ibsine4 




Figure 4.10(c) Target outsine6 and the output due to ibsine6 
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Figure 4.10(d) Target outsine7 and the output due to ibsine7 



Figure 4.10(e) Target outsine4 and the output due to insineS 



82 


4. Experiment 4 

In this experiment we investigated into the possibilities of training the 
network to recognise binary patterns obtained by reversing the bits in the 
exemplar with a probability O.i. The three exemplar sequences we chose were 
periodic sequences of total length 160. Each period had 16 values where the 
subsequence in a period was one of the rows of a 16x16 Hadamard matrix. The last 
three rows of the Hadamard matrix were used for the three exemplars respectively. 
Each of these sequences were used both as inputs and their own targets at the 
time of training the network. The figures 4.11, 4.12 and 4.13 show one period of 
each of these sequences hi, h2 and h3 respectively. The corrupted sequences hbl, 
hb2 and hb3 corresponding to hi, h2 and h3 are shown in figures 4.14, 4.15 and 
4.16 respectively. Training was carried out in two stages, first using the learning 
coefficients in table 4.2(b) and then continued using the ones in table 4.2(d). The 
weights and filter parameters obtained on training are shown in table 4.3.7 while 
the results of the experiment are shown in table 4.3.8. 

Inputs and targets: 


Cl] 

hi : 

Cl 

-1 1 -1 

-1 1 -1 1 

-1 1 -1 1 

1 -1 1 -1] 

C2] 

h2 : 

Cl 

1 -1 -1 

-1 -1 1 1 

-1 -1 1 1 

1 1 -1 -1] 

C3] 

h3 : 

Cl 

-1 -1 1 

-1 1 1 -1 

-1 1 1 -1 

1 -1 -1 1] 


Test inputs ; 

Cl] hbl 
C21 hb2 
[3] hb3 

Error at start = 479.91 


Error reduced to 13.29 
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Figure 4.12 Pattern h2 obtained from the 15th row of a 16x16 Hadamard Matrix 



Figure 4.13 Pattern h3 obtained from the 16th row of a 16x16 Hadamard Matrix 
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Figure 4.14 Pattern hbl obtained by corrupting exemplar hi. 
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Figure 4.15 Pattern hb2 obtained by corrupting exemplar h2. 



Figure 4.16 Pattern hb3 obtained by corrupting exemplar h3. 
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The error decreased drastically when learning progressed as seen above. 
Such a low value of E could be reached probably because the three exemplar 
seauences are orthogonal to one another. This is a case where target prediction 
is exceptional. On feeding the input test patterns obtained by corrupting the 
actual exemplars the net could easily identify the classes to which these belonged 
to. However, the target prediction was not as good as it was with the original 
exemplar input patterns, since the error E failed to be within the i.0'/. limit 
stipulated in the objectives. What is interesting is that the network could actually 
recognise corrupted patterns obtained by reversing the bits of the exemplar 
patterns with a probabilty of as high as 0 .4 . The prediction in this case was 
obviously poor, nevertheless the network could well recognise the corrupted inputs 
with no ambiguity at all. 


Error = 13.293022 

II 12 J1 J2 J3 J4 J5 K1 

u 1.59397 125129 47.37432 2.10657 1.48322 025149 0.83128 2.95072 

a -035616 0.16221 0.50000 0.01513 -0.02656 -0.06974 -004575 -025664 


W il 

f j 


428524 

0.91606 103820 

025921 020018 

11.45798 0.07855 


WEIGHTS 

2.78247 

028031 0.11213 029729 0.14653 

027263 0.14760 

009317 008700 008666 


Table 4.3.7 Weights and filter parameters of experiment 4 (section III) 



S.no. 

Input 

Target 

Error 

Class 

i. 

hi 

hi 

RTZ 

RT 



h2 

361.54 




h3 

375.48 


2. 

h? 

hi 

303.92 




h2 

3.50 

h2 



h3 

261.94 


3. 

h3 

hi 

289.40 




h2 

345.74 




h3 

4.68 

h3 

4. 

hbi 

hi 

33.66 

hi 



h2 

370.42 




h3 

355.01 


5. 

hb2 

hi 

335.05 




h2 

46.35 

h2 



h3 

277.54 


6. 

hb3 

hi 

287.11 




h2 

348.36 




h3 

48.66 

h3 


Table 4.3.8 Results of experiment 4 (section III) 



IV. Spatio-temporal pattern recognition 


In the experiments described so far we dealt with a single-input single- 
output network and performed purely temporal pattern recognition. In the next two 
experiments that we describe we deal with a 2-input 2-output system which 
performs spatio-temporal pattern recognition. 

1. Experiment 1 

In this experiment a network with two input and two output nodes was used. 
The learning rates and number of neurons in each layer are shown in table 4.4.1. 


layer 

Neurons 

T? 



4 

2 

0.1 

0.1 

0.1 

3 

5 

0.1 

0.1 

0.1 

2 

2 

0.1 

0.1 

0.1 

1 

-2 

- 

- 

- 


Table 4.4.1 Learning rates used in experiment 1 (section IV) 

On the lines of the experiments in section I, we examined whether the network with 
the number of elements as shown in table 4.4.1, could learn a vector sequence. We 
chose two sinusoids for the two input sequences that formed a spatio-temporal 
pattern. The targets corresponding to these two sequences in earlier experiments, 
now represented a single target pattern in this experiment as shown below. 

Input : ins 12 = (insine l,insine2) 

Target : outsl2 = <outsinei,outsine2> 

Test inputs : 

til ibsi2 = (ibsinel,ibsine2) 
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[23 ins 13 = (insine i,insine3) 

[33 ins21 = (insine2,insinei) 

Test target : outs2i = (outsine2,outsinel) 

Error at start = 82.11 
Error reduced to 27.33 

The recognition results are listed in table 4.4.2 and the final weights and filter 
parameters are shown in table 4 .4 .3 . 


S.no. 

Input 

Target 

Error 

Class 

i. 

insl2 

outs 12 

27.33 

outs 12 

2. 

ibsi2 

outsi2 

30.60 

outs 12 

3. 

insl2 

outs2i 

168.64 

NOT (outs21) 

4. 

ins 13 

outsi2 

155.64 

NOT (outsl2) 

5. 

ins2i 

outs 12 

167.03 

NOT (outs 12) 


Table 4.4.2 Results of experiment 1 (section IV) 

The entries (1) and (2) in table 4.4.2 show how well the network recognises the 
training pattern as well as the noise corrupted version of this pattern. Entry (3) 
shows that the network makes a clearout distinction between the target outs!2 = 
(outsinei,outsine2) and the test target outs21 = (outsineZ , outsinei) . Likewise, entry 
<53 shows that the network clearly discriminates between (insinei,insine2) and 
(insine2, insine 1). Thus, it preserves the spatial ordering in a spatio-temporal 
pattern. Entry (4) shows how the network can distinguish a pattern foreign to a 
class. 


As far as learning and error-minimisaton are concerned the network behaved 
in exactly the same manner in which it did in earlier experiments. The next 
experiment deals with spatio-temporal pattern recognition. 



Error = 27.328609 

u 1.59841 1.27803 2.53387 1.83778 1.48736 025121 0.83063 2.95354 

229808 

a -029532 0.50000 -029188 -0.50000 -0.01140 0.12719 0.09032 -0.38728 

-0.06803 


WEIGHTS 


W" 

1.08973 

1.15624 

125827 

126015 

1.14650 

1.14755 

1X32910 

1X39817 

Jl 

0.13689 

0.15642 

0.15697 

0.15222 

*kj 

5.33426 

424748 

4.35246 

367675 

0.07585 0.05820 

0.08270 0.04160 

0.03893 

0.04636 


0.17060 0.16107 


Table 4.4.3 Weights and filter parameters of experiment 1 (section IV) 

2. Experiment 2 

In this experiment we trained three spatio-temporal patterns and carried out 
recognition tests. The network once again had two inputs and two outputs. 
Table 4.4.1 and 4.4.4 list the values of the learning coefficients used in two stages 
of learning. At first the coefficients shown in table 4.4.1 were used to train the 
net until the network found a deep enough minimum. Then, learning was carried out 
further using the coefficients in table 4 .4 .4 . The patterns are as follows 
Inputs : 

[13 ins 12 = (insinei,insine2) 

[23 insi4 = (insinei,insine4) 


[33 ins21 = (insine2, insine 1) 



Targets : 


C13 outsl2 = <outsinel,outsine2) 

[23 outs 14 = (outsinel.,outsine4) 

C33 outs21 = <outsine2,outsinel) 

Test inputs : 

[13 ibsi2 = (ibsinel f ibsine2) 

[23 ibsl4 = (ibsinei,ibsine4) 

[33 ibs21 = (ibsine2,ibsinel) 

[43 ins24 = (insine2,insine4) 

[53 ibs24 = (ibsine2,ibsine4) 

Error at start = 238.22 
Error reduced to 46.07 

The results of the experiment are shown in table 4.4.5 and the weights and filter 
parameters obtained on training are shown in table 4.4.6. 


layer 

Neurons 

1? 


*a 

.4 

2 

0.02 

0.01 

0.001 

3 

5 

0.05 

0.02 

0.005 

2 

2 

0.1 

0.02 

0.008 

1 

Z 

_ 

- 



Table 4.4.4 Learning rates used in experiment 2 (section IV) 

V. Results on robustness 

ALFANS was found to be quite robust as illustrated by the results of the 
experiment we describe now. We trained the network using patterns of experiment 1 


in section III. The weights and filter parameters of the trained network are shown 



S.no. 

Input 

Target 

Error 

Class 

i. 

ins 12 

outs 12 
outsl4 
outs2i 

19.17 

106.89 

138.09 

outs 12 

2. 

insi4 

outs 12 
outs 14 
outs21 

120.27 

6.00 

156.92 

outs 14 

3. 

ins2i 

outs 12 
outsl4 
outs21 

135.20 

139.36 

20.90 

outs2i 

4. 

ibsi2 

outs 12 
outs 14 
outs21 

22.93 

115.94 

141.02 

outs 12 

5. 

ibsi4 

outsl2 
outs 14 
outs21 

112.08 

12.02 

151.12 

outs 14 

6. 

ibs2i 

outsi2 

outsl4 

outs2i 

134.62 

145.51 

20.90 

outs21 

7. 

ins24 

outs 12 
outs 14 
outs21 

148.87 

34.68 

128.06 

outs 14 

8 . 

ibs24 

outs 12 
outs 14 
outs21 

140.42 

40.45 

122.81 

outs!4 


Table 4.4.5 Results of experiment 2 (section IV) 


Error = 46.068199 


<•> 


a 


*8 

*ji 

\i 


1.56325 

2.06059 

-024026 

061940 

1.48766 

025044 0.82891 2.95529 

229220 






0.03812 

0.44304 

- 0.06150 

-089680 

-080036 

0.17263 0.12463 -029172 

- 0.05569 









WEIGHTS 


0.99603 

186230 


1.14510 

1.15020 


1.92182 

168761 


156612 

1.41117 

0.19135 0.17231 

0.14850 

0.16251 


0.16886 

0.15905 


3.79640 

2.72963 

0.07585 

085820 

083893 


4.91471 

4.19924 

088270 

084160 

084636 



Table 4.4.6 Weights and filter parameters of experiment 2 (section IV) 



in table 4.5.1. These weights and filter parameters were perturbed by editing them 
manually. The set of weights and parameters that resulted are shown in table 4.5.2. 
The network was then employed to recognise the test patterns of experiment i of 
section III. The resulting errors as well as the classification decisions are shown 
in table 4.5.3 along with the errors that resulted in experiment 1 of section III. 

From table 4.5.3 it can been seen that there is virtually no ambiguity in 
classifying the input even when some of the weights and filter parameters have 
been perturbed around their original values after training. Only a degradation in 
performance can be noted, in the sense that the prediction error E 


Error = 14.235181 

II 12 J1 J2 J3 J4 .IS K1 

to 1.79858 0.28189 91.3500 2.11522 1.49123 0.24862 0.82B28 2.90972 

a -0.07537 -0.5000 -0.17635 -0.05913 -0.00642 0.11017 0.06117 0.46169 


w jf 475082 

w .. -1.57750 0.45769 

Jl 

J 0.45752 0.30029 

Wi . -13.80250 0.07855 

kj 


WEIGHTS 

1.27432 

0.45962 0.20140 0.50480 0.24955 

0.46644 0 24458 

0.09317 0.08700 0.08666 


Table 4.5.1 Weights and filter parameters of experiment 1 of section III. 

between the output and the target can be seen to have increased while the error 
between the ouput and some unrelated target can be seen to have decreased. But 
clearly, these changes in error E are not large enough to influence decision 
making and jeopardize the recognizing capabilities of the system. 
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!2 »11 J2 J3 J4 jg Kj 

" 179038 °- 26189 905000 ZJ01522 1.49113 024862 0B2828 2.7097? 
a -0.07017 -0.47200 -0.17635 -006913 -000642 0.10017 006117 0.46169 


»il 

*ji 

w kj 


4.750 

-127750 0.49769 

0.35752 0.30029 

-10.80250 0.07855 


WEIGHTS 

1.37032 

0.45962 02000 0.40480 024955 

0.46644 024458 

0.09317 0.08700 0.09666 


Table 4.5.2 Weights and filter parameters obtained on perturbation. 


S.no. Input 

Target 

Error 

(section m 

(Experiment i) 

Error 

(Robustness) 

(Experiment) 

Class 

i. insine 1 

outsinel 

outsine2 

9.17 

70.59 

12.63 

50.48 

outsinel 

2 . insine2 

outsinei 

75.42 

55.18 



outsine2 

5,07 

.. 8.30 

outsine2 

3. ibsinei 

outsinel 

15.01 

14.93 

outsinei 


outsine2 

72.74 

50.16 


4. ibsine2 

outsinel 

76.56 

55.12 



outsine2 

9.86 ..... _ 

10.46 

outsine2 

5. insineS 

outsinel 

76.79 

55.95 



outsine2 

616 

8.89 

PUtrSineg 


Table 4.5.3 Results of experiment on robustness 
VI. Results on convergence 

Some experiments were performed to examine convergence of the learning 


algorithm. Empirically, the algorithm can be considered to be convergent if it 



leads to stable values of weights and filter parameters; values that change very 
little when learning is continued further. 

In experiment i of section III we carried out learning, initially with the 
rates shown in table 4.2(b). The error at start was 84.605 which reduced to 17.463 
after 114 iterations. Thereafter, error increased drastically. This happened 
probably, because the weights and parameters had grown to such high values that 
the input net(n) to the non-linearity in each neural unit had increased so much in 
magnitude as to cause the neural units to go into saturation, thereby causing the 
neural outputs to switch between the -1 and +1 limits. Therefore, in the back- 
propagation phase the weights and parameters did not adjust finely. Instead they 
changed drastically and made the algorithm show divergent behaviour. 

After the 114th learning cycle we reduced the learning coefficients to 
those shown in table 4.2 (c) and carried on with the learning procedure. The system 
encountered another minimum and E reduced further down to 14.246 after another 
163 iterations and once again the error started increasing. Upon further reducing 
the learning coefficients to those in table 4.2 (d) the network cruised slowly and 
the error reduced to 14.235 and varied very little thereafter. Plots of error E 
versus learning cycles for various cases of learning coefficients in a number of 
experiments are shown in figures 4.17(a) through (f). The weights and filter 
parameters for experiment 1 of section III, plotted against learning cycles can be 
seen in figures 4.18(a) and (b), 4.19 and 4.20. The trend seen here is what has been 
exhibited in almost all experiments that we have described earlier. 

Our conclusion based on the empirical observations we collected is that the 
network weights and parameters attain stable values at sufficiently low learning 
rates and that the algorithm displays convergent behaviour. However, if the 
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UCARHHC CYCLE 

(c) 

Figure 4.17 Error E vs Learning cycle, (a) lexpt.l sect. II] E varies little at low 
learning rates. Weights and parameters converge, (b) [expt.l sect. Ill] On 
switching to low rates, weights and parameters converge, (c) lexpt.4 sect. Ill] 
Error E reduces steadily at low learning rates. 





o 200 400 

learning cycle 

(f) 

Figure 4.17 Error E vs Learning cycle, (d) [expt.3 sect.lll] At high learning rates, 
error blows up after a minimum, (e) [expt.5 sect.lll At large rates of learning, 
error blows up after reaching a minimum twice, (f) [sect. VI] Error reaches a new 
minimum. Perturbed weights and parameters take new values. System is seeking a 
second minimum at low learning rates. 
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4.18(a) Weights vs Learning cycle in experiment 1 of section III. 
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Figure 4.18(b) Weights vs Learning cycle in experiment 1 of section 111, 
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Figure 4.19 The decay rates a vs learning cycle in experiment 1 of section III. 
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Figure 4.20 The parameters u vs learning cycle in experiment 1 of section 
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learning coefficients are high then the network exhibits saturaton after going 
through a deep enough minimum in the error E and the algorithm exhibits 
divergence . 

That the algorithm converges monotonically cannot be true because the 
system actually jumps from one spurious minimum to another till it lands into a 
deep enough minimum. If the learning rate coefficients are small then the algorithm 
does not move the system out of the minimum and the algorithm displays convergent 
characteristics . What is lacking is any sort of theoretical analysis that can 
acertain convergence or divergence of this algorithm. This is because of the 
highly non-linear nature of the neural system. 

In another experiment, we used the weights and patterns associated with 
experiment i of section III and fed the inputs insinei and insine2 to the network. 
The generated outputs y^ and y 2 were then made the targets and the error was 
obviously zero. Next, we used the perturbed weights used in section V (see table 
4.5.2) and carried on with the learning process with y^ and y 2 as the targets. The 
results obtained are as shown below 
Inputs : 

E13 insinei C23 insine2 

Targets : 

C13 y ± £23 y 2 

Error at start = 10.3804 
Error reduced steadily to 4.6239 

The system reached an error minimum and the weights and filter parameters 
stabilised at their new values as shown in table 4.6.1. This clearly shows that the 
network has multiple choices of weights and parameters which minimise the error E 
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although at the minimum the error E may not be the same on all occasions. In this 
particular exrperiment the error could never have reached zero because some 
initial error always exists in predicting the target pattern at n=0 and this 
contributes a non-zero part to the total error E. 


Error = 4.623922 


11 12 J1 J2 J3 J4 JS K1 

u i .92461 025811 90.70726 2.01551 1.49116 024849 0.82800 2.70192 

a -0.16523 -050000 -0.48256 -006953 -0.00632 0.10128 0.06209 0.45425 


401934 

I 

l> .. -1.19193 0.46377 

II 

d 005686 0.30055 

u» . -10.71904 0.07855 

kj 


WEIGHTS 

103039 

0.45906 020022 0.40415 024980 

0.46575 024485 

009317 0.08700 009666 


Table 4.6.1 New values of weights and filter parameters obtained in the above 
experiment 

With these results we conclude this chapter, having presented a 
considerable number of cases to bring out the various idiosynchracies of the 
network and the associated learning algorithm alongwith the results that make 
evident the capacity of the system to perform spatio-temporal pattern recognition. 



Chapter 5 

Conclusion and Discussion 


In this thesis various aspects of spatio-temporal pattern recognition have 
been considered along with discussion on some approaches that researchers have 
taken to perform this task. Chapter 2 discusses some of the networks that 
researchers have proposed for speech recognition and text to phoneme 
transcription. This chapter also includes a discussion on outstar based networks 
which can be used for generating complex spatio-temporal patterns useful in tasks 
such as motion control of robots . There is also a discussion on how self- 
organising networks can be used for speech segmentation and recognition. 

Some ideas that led to the design of a feed-forward back-propagation 
network for spatio-temporal pattern recognition have been presented. Based on 
these ideas a linear filter artificial neuron system has been proposed in 
chapter 3 and a learning algorithm has been derived to train this network. 
Conditions on the stability of various filters used in this network have also been 
noted. Results of computer simulations to test this network have been presented in 
chapter 4. 

I. Comments and suggestions for future work 

There are some critical comments to be made on ALFANS. In this system the 
output neural units perform some kind of signal synthesis. The inputs to these 
units are sinusoids or decaying sinusoids, or some such signals and the weights 
connecting the output units to the previous layer, act like the synthesis 



coefficients. The output signal so synthesized, is expected to predict the target 


with fair accuracy. Based on our simulations we observe that the prediction 
performance of the network is not upto the mark in most cases. One possible 
suggestion to improve the performance is to remove the filters in the output units 
and retain the standard neuron that is normally used in feed forward networks for 
static pattern recognition. This essentially amounts to chosing an output filter 
with a unit impulse for its impulse response. This way the low pass nature of the 
output filter that is now diminishing the high frequency content of the output 
signal will be eliminated. An increase in high frequency content of the synthesized 
output is very likely to enhance the perfomance of the network in predicting the 
target waveform. 

One other suggestion is that an empirical study of this network be carried 
out using various kinds of linear filters in order to examine as to what type of 
filters are most appropriate for a given set of patterns to be trained. For 
instance, it should seem appropriate to use first order filters to train decaying 
exponentials, while a resonator could possibly enhance the network's performance 
on recognition of sinusoids. 

There is an important observation to make regarding the learning rate 
coefficients. It is very clear that the algorithm is not convergent for large values 
of learning rate coefficients although it is seen to converge for small rates. 
However, large rates allow the system to quickly bypass local minima and descend 
along a deep minimum, while small values of learning coefficients cause the system 
to glide slowly and increase the possibility of the system getting stuck in a 
spurious local minimum. To get the advantages of these two cases while eliminating 
their disavantages one could consider the possibility of reducing learning rates 



based on the decrease in error from the starting value. For example one could 
consider making 
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1 

|jrSt3irt jrcurrentjd 


( 5 . 1 ) 


where E start is the error at start, £ current is the error after the current 
iteration, and T? next is the value of 7? for the next iteration. 

Another option is to use low value learning rate coefficients and include 
momentum terms when calculating the increments. This has been highly successful 
in case of the standard back-propagation network. It is expected that addition of 
the momentum term will increase the pace of learning and bypass spurious minima 
even in this algorithm. This has to be examined by carrying out some simulation 
experiments . 

In this thesis we have dealt with sinusoidal as well as binary patterns. One 
could possibly take up speech recognition using this system in a future endeavour. 
A short-time Fourier transform can be treated as a spatio-temporal pattern and 
speech recognition based on this can be performed. One could also feed sampled 
speech directly to this system without any pre-processing and train the network to 
perform temporal pattern recognition. One other application domain for ALFANS, 
besides speech processing, is robotics where it can be used to recognise spatio- 
temporal signals generated by the sensory apparatus of a robot. Sensing moving 
pictures and temporal changes in tactical stimulii are some important areas where 
ALFANS can possibly, be employed. 

The present scenario in pattern recognition, control and robotics indicates 
that much work needs to be done in spatio-temporal pattern analysis and 
recognition, and this thesis is being concluded with the view that neural networks 
will become the means to perform this task with efficiency, reliability and ease. 



Appendix A 

Jury’s Test 


Jury's test states the conditions under which a filter H(z) = > with 

N 

D<z> - biz 0 " 1 (A.i> 

to 

is stable. 

Using the coefficients of DCz) we construct the following array 


Jury's Table 


ROW 



COEFFICIENTS 

i 

b 0 

bi 

b N 


b N 

b w -i 

... b 0 

2 

Co 

Ci 

. . - C w -i 


c N-i 

c N-2 

c 0 

3 

do 

di 

. . . d w -a 


d w -2 

d N -3 

d 0 

. . - 

. . . 

. . . 

... ... 

2N-3 

r 0 



where 




C; = 

1 b 0 b N -;| 


for i=0 to N-i 


| bw b; | 


(A .2) 

d; = 

1 Co C N -i- 
| C N-i c i 

il 

1 

for i=0 to N-2 

and so on 

until the 

row 2N- 

•3 which has just 3 elements in it. Jury's stability 


criterion then, states that the filter H(z) is stable iff the following are 


satisfied. 
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(i) D(i) > 0 
<ii> <-i) N D(-i) > 0 

(iii> b 0 > |b w |, 

Cq ) | c N-i | / 

do > | dw-2 I' 

r O > I ^2 ! 


(A .3.1) 
(A .3 .2) 
(A .3 .3) 



Appendix B 
Instar and Outstar 


I. Introduction 

Instar and Outstar configuration along with the associated training rules 
were first proposed by Grossberg [163. The instar, as shown in fig. E.i. is a 
neurode fed by a number of inputs taken through set of synaptic weights. It also 
has a triggering output which may be connected to other neurodes. The outstar, as 
shown in fig. E.2 is a neurode with many outputs each of which is driven through a 
synaptic weight. It has a triggering input which allows the other neurodes to 
trigger it. In fact in any neural network, every neurode can be looked upon as the 
focus of an instar and an outstar since each neurode has many inputs and at the 
same time its output fans out to many other neurodes. Thus, a neural network can 
be considered to be a vast array of neurodes interconnected amongst themselves, 
with the inwardly feeding inputs of each instar arising from the outwardly 
directed outputs of some outstars. 

II. Instar training rule 

An instar is trained to respond to a specific input vector X only. Training 
proceeds in a way so as to adjust the input weight vector to be like the specified 
input vector X. The instar output is calculated to be the linear combination of the 
inputs. Thus, one can see that the calculation produces the dot product of the 
weight vector with an input vector applied at a given time which is a measure of 
similarity between the two, in case both of them are normalized vectors. The 
training rule is such that in the testing phase a trained instar comes to respond 



• • 



most strongly to the input pattern for which it was trained. The weight adjustment 


toe 


rule is given below. 

Wjk + ^ = + aCxj - w^l <B.i) 

where w^t is the weight from input Xj after learning iteration t, and a is the 
training rate coefficient which is chosen to be around O.i and is reduced 
gradually as training progresses. 

III. Outstar training rule 

While the instar performs recognition the outstar generates the desired 
excitation pattern to the other neurons when it fires. The outstar training rule 
adjusts the synaptic weights to be like a desired target vector. The training rule 
is as follows. 

Wj t+i = w > + ec Uj - Wj l ] (B.2) 

where wA is the weight at the output y • after learning iteration t, and fc is 
the training rate coefficient which is chosen to be around i and is gradually 
reduced to zero as training progresses. 

Both instar and outstar are trained over a set of training vectors 
belonging to a class so that the instar weights adjust themselves to represent an 
average (in some sense) of all the patterns belonging to that class and the 
outstar can converge to the exemplar pattern when the network is presented with 
distorted patterns. By including a non-linearity in series with the outstar outputs 
one can actually train the system to recognise and generate more than just one 
class of patterns. The instar and outstar are discussed in the references CiiX 


Ei61 and [541. 



Appendix C 
Neocognitron 


I. Introduction 

Neocognitron is a self -organising network proposed by Fukushima and Miyake 
C141 [153, which is tolerant of deformations as well as shifts in position of the 
impressed input pattern. This network belongs to a class of competitive learning 
networks and is an advancement over an earlier version called cognitron which was 
not shift-tolerant. Repetitive presentation of the input patterns causes the 
network to self-organise and after sufficient progress the network acquires the 
ability to classify and correctly recognise these patterns or their variants by 
perceiving the differences in their shapes. 

II. Cells in the neocognitron 

The neocognitron is heirarchically organised as shown in figure C.l . It has 
a number of modules with the innermost one composed of "gnostic cells". Each 
module has four layers, each carrying a different type of cell or neuron. Two of 
these layers, namely Us and Uc carrying the S-cells (Us cells) and the C-cells (Uc 
cells) respectively, are further divided into a number of planes each of which is 
an array of cells. The other two layers carry the Vs and Vc type cells 
respectively. 

The input-output description of these cells follows. Let u(n), n=i to N be 
the excitatory inputs and v be the inhibitory input to an S-cell. The output w 



iiO 



Figure C.l Schematic diagram illustrating the interconnections between layers in 
the neocognitron (Reproduced from [14]}. 



Figure C.2 Input-to-output characteristics of an S-cell: A typical example of the 
cells employed in the neocognitron (Reproduced from [14]) 




of the S-cell is defined by 
N 

- ' a(n)u(n) 


w = <f>\ 


1 + 


n=l 


- i 


i+ bv 


(C.i) 


where a(n> and b represent the excitatory and inhibitory interconnection 

coefficients respectively and the function <pl 3 is given by 

tfilxl - / x ^ 0) (£ 2) 

v | 0 (x < 0) 

The output of a C-cell is Quite similar except that it uses a 3 function instead 
of <p[ 3, and is given by 


Wx3 - \ 


x 

a+x 

0 


(x h 0) 
(x < 0) 


<C.3> 


The parameter a is a positive constant which 
saturation of the output. The Vc cell has inputs u(n), 
given by 


w = 


A 


N 


c(n)u(n) 


n=l 


determines the degree of 
n = i to N and its output is 

(C.4) 


The Vs-cell input-output relation is quite similar to 
neocognitron the only the S-cell connection strengths 
adjust during self-organisation. 


that of the Vc-cell. In the 
a< ) and b are allowed to 


III. Neocognitron structure 

Now we describe the process of updating the cells in the testing phase. The 
notation u sl <k},n) will be used to represent the output of an S-cell in the Iq-th 
S-plane in the 1-th module, where n is the position vector of the cell in that 



Diane. The C-cell output is denoted in a similar manner. The incut is presented to 
the Uc-cells of the C-layer of the module number 0 . The Uc cells of this module 
are first updated. Then the Vo cells of that module are updated. The updated 
outputs from these Uc and Vc cells now serve as inputs for updating the Us cells 
of the next module. The outputs from these Us cells are used as inputs to update 
the Vs cells of the same module. The updated outputs from the Us and Vs cells of 
that module are used as inputs to the Uc cells of the module in the same manner in 
which the input picture is impressed on the Uc cells of module number 0 . The Uc 
cell outputs are used to update the Vc cell outputs of that module which serve as 
inputs to the next module. This is carried out sequentially from the outermost 
module, namely module D upto the innermost module where the Uc cells are arranged 
in K planes each holding exactly one cell. These cells are representative of the 
gnostic cells in the human brain. Each of these gnostic cells represents exactly 
one class of input patterns and fires when a member of that class is impressed on 
the input layer. The constitution of a class is decided entirely by the structure 
and number of cells in the network, the scheme for self-organisation, the threshold 
parameters and learning rates, as well as, the initial choice of weights which are 
to be chosen randomly subject to some constraints. Thus, the maximum number of 
classes of input patterns can be only as many as the number of gnostic cells. 


The update equations are as follows. The inhibitory cell v c i_j.(n) output is 
given by 

I ^1-i 

v 0 j_<(n) = i! S S Ci 4(D)u 2 ^i_-i(k 1 _i,n+p) (C.5) 


jvv = * | Sc, l( p>u 2 c , 1 <k, 1 ,n + p> 

Vi=i peSj 


The values of the fixed interconnection weights Cj_^<p) are chosen so as to 
decrease monotonically with respect to |p| and to satisfy 
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(C.6) 


k l-i = l peSi 

The size of the connecting! area S} is chosen small in the first module and 
increases as the depth 1 is increased. The output of a Us cell of the kj-th S- 
plane in module 1 is given by 



u sl (kpn> 


K l-i 




ps Si 

1 + i.rjW^l-iW 


(C.7) 


where <p[ 3 is as defined in (. 02 ). In case of l=i ^l-i^l-i-Tv stands for ug(n) 
and Kq= 1. Parameter r^ in (C.7) controls the intensity of inhibition. The Vs cells 
are updated next according to the following rule 


Kl 

v sl Cn) = A 22 22 d l (p)u *l-i <k l' n+o) 

kj = 1 p e D} 


The Uc cells are the last to be updated and their input-output relationship is 
described by 


u cl (ki,n> 


«- 


i + 


^ di<p)u £l _ i <ki,n+p) 

psDj 


- i 


(C.9) 


I i+ v sl <n) I 

where 0C 3 is as defined in (C.3). The fixed interconnection strengths djfo) are 
chosen exactly like Cj(p) and once again the connecting area Dj is set small in 
the first module and is increased in the inner layers. 



IV. Self-organisation in neocognitron 


In the neocognitron self-organisation is via unsuoervised learning in which 
the network is repeatedly presented with patterns at the input. After a pattern 
presentation the cells in all the modules are updated according to the rules 
specified in the section III. Then, the variable interconnections a< ) and b are 
adjusted as per the following scheme. Firstly, several "representative" S-cells are 
chosen from among all the S-cells that yield a large output after being updated 
but the number of representatives is so restricted that more than one must not be 
chosen from any single S-plane in a module. The details of the procedure for 
choosing the representative cells are given in [143. For a representative S-cell. 
only those input interconnections which see a non-zero incoming signal are 
reinforced so that the cell becomes selectively responsive to the stimulus feature 
that is seen then. All the other S-cells in the S-plane, from which the 
representative is chosen get their input interconnection strengths reinforced by 

A A 

the same amounts as those for their representative. Let cell u s ^(kj,n> be a 
representative. Then, the incoming interconnection strengths are reinforced 
according to the rule given below 

A A 

^*1^1-1 jPj^i) = q^C2_^<p>u c ]_j_(ki_i,n+p> (C.iO) 

4bj(kj) = q^v c i_jTn) (C.li) 

where qj is a positive constant that determines the speed of increment. The cells 
in the S-plane from which no representative comes forth do not have their input 
interconnections varied at all. 

An explanation for the specific input-output relations for various cell 
types and the updating rules for interconnection strengths mentioned above along 
with an extensive description of the neocognitron behaviour are contained in the 
original paper by Fukushima and Miyake Ci43 published in 1962. 
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